Oct 31 00:55:14.066749 ip-10-0-130-39 systemd[1]: Starting Kubernetes Kubelet... Oct 31 00:55:14.490843 ip-10-0-130-39 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:55:14.490843 ip-10-0-130-39 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 00:55:14.490843 ip-10-0-130-39 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:55:14.490843 ip-10-0-130-39 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:55:14.495682 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495591 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:55:14.495682 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495681 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:55:14.495682 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495685 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:55:14.495786 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495689 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:55:14.495786 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495692 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:55:14.495786 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495695 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:55:14.495786 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495698 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:55:14.495786 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495701 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:55:14.495786 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495703 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:55:14.495786 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495706 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:55:14.495786 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495708 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:55:14.495786 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495711 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:55:14.495786 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495713 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:55:14.495786 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495716 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:55:14.495786 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495718 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:55:14.495786 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495721 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:55:14.495786 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495724 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:55:14.495786 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495726 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:55:14.495786 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495729 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:55:14.495786 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495731 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:55:14.495786 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495734 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:55:14.495786 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495737 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:55:14.495786 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495741 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:55:14.496249 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495743 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:55:14.496249 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495746 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:55:14.496249 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495749 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:55:14.496249 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495751 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:55:14.496249 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495753 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:55:14.496249 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495756 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:55:14.496249 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495759 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:55:14.496249 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495761 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:55:14.496249 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495764 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:55:14.496249 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495766 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:55:14.496249 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495768 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:55:14.496249 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495771 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:55:14.496249 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495774 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:55:14.496249 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495777 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:55:14.496249 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495783 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:55:14.496249 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495788 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:55:14.496249 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495791 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:55:14.496249 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495794 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:55:14.496249 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495797 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:55:14.496709 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495800 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:55:14.496709 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495803 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:55:14.496709 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495805 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:55:14.496709 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495808 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:55:14.496709 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495811 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:55:14.496709 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495814 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:55:14.496709 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495816 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:55:14.496709 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495819 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:55:14.496709 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495821 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:55:14.496709 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495824 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:55:14.496709 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495828 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:55:14.496709 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495831 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:55:14.496709 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495833 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:55:14.496709 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495837 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:55:14.496709 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495841 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:55:14.496709 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495844 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:55:14.496709 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495847 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:55:14.496709 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495850 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:55:14.496709 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495853 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:55:14.497198 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495856 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:55:14.497198 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495858 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:55:14.497198 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495861 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:55:14.497198 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495864 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:55:14.497198 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495866 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:55:14.497198 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495869 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:55:14.497198 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495872 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:55:14.497198 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495875 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:55:14.497198 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495878 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:55:14.497198 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495880 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:55:14.497198 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495883 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:55:14.497198 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495886 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:55:14.497198 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495888 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:55:14.497198 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495891 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:55:14.497198 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495894 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:55:14.497198 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495896 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:55:14.497198 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495899 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:55:14.497198 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495901 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:55:14.497198 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495904 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:55:14.497685 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495906 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:55:14.497685 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495909 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:55:14.497685 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495911 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:55:14.497685 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495914 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:55:14.497685 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495917 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:55:14.497685 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495920 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:55:14.497685 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495923 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:55:14.497685 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495926 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:55:14.497685 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495929 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:55:14.497685 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495932 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:55:14.497685 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495934 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:55:14.497685 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495937 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:55:14.497685 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495939 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:55:14.497685 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495942 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:55:14.497685 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495945 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:55:14.497685 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495947 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:55:14.497685 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495950 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:55:14.497685 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495952 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:55:14.497685 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495956 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:55:14.498134 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495959 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:55:14.498134 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495961 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:55:14.498134 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495964 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:55:14.498134 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495967 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:55:14.498134 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495969 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:55:14.498134 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495973 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:55:14.498134 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.495976 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:55:14.498134 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496853 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:55:14.498134 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496862 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:55:14.498134 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496866 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:55:14.498134 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496869 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:55:14.498134 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496872 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:55:14.498134 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496875 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:55:14.498134 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496878 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:55:14.498134 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496881 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:55:14.498134 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496883 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:55:14.498134 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496886 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:55:14.498134 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496889 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:55:14.498134 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496891 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:55:14.498634 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496895 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:55:14.498634 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496898 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:55:14.498634 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496902 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:55:14.498634 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496904 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:55:14.498634 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496907 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:55:14.498634 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496910 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:55:14.498634 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496912 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:55:14.498634 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496915 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:55:14.498634 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496917 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:55:14.498634 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496920 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:55:14.498634 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496923 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:55:14.498634 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496925 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:55:14.498634 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496928 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:55:14.498634 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496930 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:55:14.498634 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496933 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:55:14.498634 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496935 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:55:14.498634 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496938 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:55:14.498634 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496940 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:55:14.498634 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496943 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:55:14.498634 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496945 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:55:14.499106 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496950 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:55:14.499106 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496953 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:55:14.499106 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496957 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:55:14.499106 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496960 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:55:14.499106 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496963 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:55:14.499106 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496967 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:55:14.499106 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496970 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:55:14.499106 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496973 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:55:14.499106 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496975 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:55:14.499106 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496978 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:55:14.499106 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496981 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:55:14.499106 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496983 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:55:14.499106 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496986 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:55:14.499106 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496988 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:55:14.499106 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496991 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:55:14.499106 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496994 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:55:14.499106 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496996 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:55:14.499106 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.496999 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:55:14.499106 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497001 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:55:14.499568 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497004 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:55:14.499568 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497007 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:55:14.499568 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497009 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:55:14.499568 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497012 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:55:14.499568 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497014 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:55:14.499568 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497017 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:55:14.499568 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497020 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:55:14.499568 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497022 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:55:14.499568 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497025 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:55:14.499568 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497027 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:55:14.499568 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497029 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:55:14.499568 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497032 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:55:14.499568 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497034 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:55:14.499568 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497037 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:55:14.499568 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497039 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:55:14.499568 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497042 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:55:14.499568 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497045 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:55:14.499568 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497047 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:55:14.499568 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497049 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:55:14.500047 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497052 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:55:14.500047 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497055 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:55:14.500047 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497058 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:55:14.500047 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497061 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:55:14.500047 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497063 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:55:14.500047 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497067 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:55:14.500047 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497071 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:55:14.500047 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497074 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:55:14.500047 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497077 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:55:14.500047 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497080 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:55:14.500047 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497083 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:55:14.500047 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497085 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:55:14.500047 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497088 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:55:14.500047 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497091 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:55:14.500047 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497094 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:55:14.500047 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497096 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:55:14.500047 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497099 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:55:14.500047 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497101 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:55:14.500047 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497104 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:55:14.500512 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497107 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:55:14.500512 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497109 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:55:14.500512 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497112 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:55:14.500512 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497114 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:55:14.500512 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497117 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:55:14.500512 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497119 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:55:14.500512 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497122 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:55:14.500512 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497125 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:55:14.500512 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497128 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:55:14.500512 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497131 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:55:14.500512 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497133 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:55:14.500512 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497136 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:55:14.500512 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497138 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:55:14.500512 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497141 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:55:14.500512 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497143 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:55:14.500512 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497147 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:55:14.500512 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.497149 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:55:14.500512 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498016 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:55:14.500512 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498022 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:55:14.500512 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498025 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:55:14.500993 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498028 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:55:14.500993 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498031 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:55:14.500993 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498034 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:55:14.500993 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498036 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:55:14.500993 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498039 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:55:14.500993 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498042 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:55:14.500993 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498045 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:55:14.500993 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498047 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:55:14.500993 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498050 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:55:14.500993 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498052 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:55:14.500993 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498055 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:55:14.500993 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498059 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:55:14.500993 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498061 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:55:14.500993 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498064 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:55:14.500993 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498067 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:55:14.500993 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498069 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:55:14.500993 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498072 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:55:14.500993 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498074 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:55:14.500993 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498078 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:55:14.501466 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498080 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:55:14.501466 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498083 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:55:14.501466 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498086 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:55:14.501466 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498089 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:55:14.501466 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498092 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:55:14.501466 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498094 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:55:14.501466 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498097 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:55:14.501466 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498099 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:55:14.501466 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498102 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:55:14.501466 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498105 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:55:14.501466 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498107 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:55:14.501466 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498110 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:55:14.501466 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498112 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:55:14.501466 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498114 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:55:14.501466 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498117 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:55:14.501466 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498120 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:55:14.501466 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498123 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:55:14.501466 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498125 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:55:14.501466 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498127 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:55:14.501466 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498130 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:55:14.502029 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498132 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:55:14.502029 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498135 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:55:14.502029 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498138 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:55:14.502029 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498140 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:55:14.502029 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498143 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:55:14.502029 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498145 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:55:14.502029 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498148 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:55:14.502029 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498150 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:55:14.502029 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498153 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:55:14.502029 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498156 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:55:14.502029 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498158 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:55:14.502029 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498163 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:55:14.502029 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498166 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:55:14.502029 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498168 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:55:14.502029 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498171 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:55:14.502029 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498174 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:55:14.502029 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498177 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:55:14.502029 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498179 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:55:14.502029 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498182 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:55:14.502029 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498184 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:55:14.502540 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498187 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:55:14.502540 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498189 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:55:14.502540 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498192 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:55:14.502540 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498194 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:55:14.502540 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498197 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:55:14.502540 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498199 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:55:14.502540 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498202 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:55:14.502540 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498205 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:55:14.502540 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498208 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:55:14.502540 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498212 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:55:14.502540 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498215 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:55:14.502540 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498217 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:55:14.502540 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498220 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:55:14.502540 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498223 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:55:14.502540 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498225 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:55:14.502540 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498228 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:55:14.502540 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498230 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:55:14.502540 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498233 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:55:14.502540 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498235 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:55:14.502979 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498239 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:55:14.502979 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498243 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:55:14.502979 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498247 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:55:14.502979 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498250 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:55:14.502979 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498254 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:55:14.502979 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498257 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:55:14.502979 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498259 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:55:14.502979 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498262 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:55:14.502979 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498264 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:55:14.502979 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498267 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:55:14.502979 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498270 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:55:14.502979 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498272 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:55:14.502979 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498275 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:55:14.502979 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498277 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:55:14.502979 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498280 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:55:14.502979 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498283 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:55:14.502979 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498285 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:55:14.502979 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498288 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:55:14.502979 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498290 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:55:14.503432 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498293 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:55:14.503432 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498295 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:55:14.503432 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498298 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:55:14.503432 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498301 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:55:14.503432 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498304 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:55:14.503432 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.498307 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:55:14.503432 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499372 2576 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 00:55:14.503432 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499389 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 00:55:14.503432 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499396 2576 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 00:55:14.503432 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499401 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 00:55:14.503432 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499406 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 00:55:14.503432 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499410 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 00:55:14.503432 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499426 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 00:55:14.503432 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499432 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 00:55:14.503432 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499435 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 00:55:14.503432 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499438 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 00:55:14.503432 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499442 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 00:55:14.503432 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499448 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 00:55:14.503432 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499451 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 00:55:14.503432 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499454 2576 flags.go:64] FLAG: --cgroup-root="" Oct 31 00:55:14.503432 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499457 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 00:55:14.503432 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499460 2576 flags.go:64] FLAG: --client-ca-file="" Oct 31 00:55:14.503949 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499463 2576 flags.go:64] FLAG: --cloud-provider="external" Oct 31 00:55:14.503949 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499466 2576 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 00:55:14.503949 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499470 2576 flags.go:64] FLAG: --cluster-domain="" Oct 31 00:55:14.503949 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499473 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 00:55:14.503949 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499476 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 00:55:14.503949 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499479 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 00:55:14.503949 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499482 2576 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 00:55:14.503949 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499487 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 00:55:14.503949 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499490 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 00:55:14.503949 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499493 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 00:55:14.503949 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499497 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 00:55:14.503949 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499500 2576 flags.go:64] FLAG: --contention-profiling="false" Oct 31 00:55:14.503949 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499504 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 00:55:14.503949 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499507 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 00:55:14.503949 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499511 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 00:55:14.503949 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499514 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 00:55:14.503949 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499519 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 00:55:14.503949 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499522 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 00:55:14.503949 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499525 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 00:55:14.503949 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499529 2576 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 00:55:14.503949 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499532 2576 flags.go:64] FLAG: --enable-server="true" Oct 31 00:55:14.503949 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499536 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 00:55:14.503949 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499540 2576 flags.go:64] FLAG: --event-burst="100" Oct 31 00:55:14.503949 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499544 2576 flags.go:64] FLAG: --event-qps="50" Oct 31 00:55:14.503949 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499547 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 00:55:14.504646 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499550 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 00:55:14.504646 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499553 2576 flags.go:64] FLAG: --eviction-hard="" Oct 31 00:55:14.504646 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499559 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 00:55:14.504646 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499562 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 00:55:14.504646 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499566 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 00:55:14.504646 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499569 2576 flags.go:64] FLAG: --eviction-soft="" Oct 31 00:55:14.504646 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499572 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 00:55:14.504646 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499575 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 00:55:14.504646 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499579 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 00:55:14.504646 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499582 2576 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 00:55:14.504646 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499585 2576 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 00:55:14.504646 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499588 2576 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 00:55:14.504646 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499591 2576 flags.go:64] FLAG: --feature-gates="" Oct 31 00:55:14.504646 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499595 2576 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 00:55:14.504646 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499598 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 00:55:14.504646 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499601 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 00:55:14.504646 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499605 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 00:55:14.504646 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499612 2576 flags.go:64] FLAG: --healthz-port="10248" Oct 31 00:55:14.504646 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499616 2576 flags.go:64] FLAG: --help="false" Oct 31 00:55:14.504646 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499619 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-130-39.us-east-2.compute.internal" Oct 31 00:55:14.504646 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499622 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 00:55:14.504646 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499625 2576 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 00:55:14.504646 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499629 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 00:55:14.505197 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499632 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 00:55:14.505197 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499636 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 00:55:14.505197 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499639 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 00:55:14.505197 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499642 2576 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 00:55:14.505197 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499645 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 00:55:14.505197 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499648 2576 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 00:55:14.505197 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499651 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 00:55:14.505197 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499654 2576 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 00:55:14.505197 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499657 2576 flags.go:64] FLAG: --kube-reserved="" Oct 31 00:55:14.505197 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499660 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 00:55:14.505197 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499663 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 00:55:14.505197 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499667 2576 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 00:55:14.505197 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499670 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 00:55:14.505197 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499673 2576 flags.go:64] FLAG: --lock-file="" Oct 31 00:55:14.505197 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499676 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 00:55:14.505197 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499679 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 00:55:14.505197 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499682 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 00:55:14.505197 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499688 2576 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 00:55:14.505197 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499691 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 00:55:14.505197 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499694 2576 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 00:55:14.505197 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499698 2576 flags.go:64] FLAG: --logging-format="text" Oct 31 00:55:14.505197 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499701 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 00:55:14.505197 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499704 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 00:55:14.505197 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499707 2576 flags.go:64] FLAG: --manifest-url="" Oct 31 00:55:14.505795 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499710 2576 flags.go:64] FLAG: --manifest-url-header="" Oct 31 00:55:14.505795 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499715 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 00:55:14.505795 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499719 2576 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 00:55:14.505795 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499723 2576 flags.go:64] FLAG: --max-pods="110" Oct 31 00:55:14.505795 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499726 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 00:55:14.505795 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499730 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 00:55:14.505795 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499733 2576 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 00:55:14.505795 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499736 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 00:55:14.505795 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499739 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 00:55:14.505795 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499743 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 00:55:14.505795 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499746 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 00:55:14.505795 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499754 2576 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 00:55:14.505795 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499758 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 00:55:14.505795 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499761 2576 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 00:55:14.505795 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499764 2576 flags.go:64] FLAG: --pod-cidr="" Oct 31 00:55:14.505795 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499767 2576 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 00:55:14.505795 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499770 2576 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 00:55:14.505795 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499773 2576 flags.go:64] FLAG: --pods-per-core="0" Oct 31 00:55:14.505795 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499776 2576 flags.go:64] FLAG: --port="10250" Oct 31 00:55:14.505795 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499781 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 00:55:14.505795 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499784 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0dd13a34fbb3e16a9" Oct 31 00:55:14.505795 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499788 2576 flags.go:64] FLAG: --qos-reserved="" Oct 31 00:55:14.505795 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499791 2576 flags.go:64] FLAG: --read-only-port="10255" Oct 31 00:55:14.505795 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499794 2576 flags.go:64] FLAG: --register-node="true" Oct 31 00:55:14.505795 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499797 2576 flags.go:64] FLAG: --register-with-taints="" Oct 31 00:55:14.505795 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499801 2576 flags.go:64] FLAG: --registry-burst="10" Oct 31 00:55:14.506411 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499804 2576 flags.go:64] FLAG: --registry-qps="5" Oct 31 00:55:14.506411 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499807 2576 flags.go:64] FLAG: --reserved-cpus="" Oct 31 00:55:14.506411 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499810 2576 flags.go:64] FLAG: --reserved-memory="" Oct 31 00:55:14.506411 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499814 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 00:55:14.506411 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499817 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 00:55:14.506411 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499820 2576 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 00:55:14.506411 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499823 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 00:55:14.506411 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499826 2576 flags.go:64] FLAG: --runonce="false" Oct 31 00:55:14.506411 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499830 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 00:55:14.506411 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499834 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 00:55:14.506411 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499837 2576 flags.go:64] FLAG: --seccomp-default="false" Oct 31 00:55:14.506411 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499840 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 00:55:14.506411 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499842 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 00:55:14.506411 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499846 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 00:55:14.506411 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499849 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 00:55:14.506411 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499853 2576 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 00:55:14.506411 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499856 2576 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 00:55:14.506411 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499858 2576 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 00:55:14.506411 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499861 2576 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 00:55:14.506411 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499865 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 00:55:14.506411 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499868 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 00:55:14.506411 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499871 2576 flags.go:64] FLAG: --system-cgroups="" Oct 31 00:55:14.506411 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499874 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 00:55:14.506411 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499880 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 00:55:14.506411 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499883 2576 flags.go:64] FLAG: --tls-cert-file="" Oct 31 00:55:14.507411 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499887 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 00:55:14.507411 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499891 2576 flags.go:64] FLAG: --tls-min-version="" Oct 31 00:55:14.507411 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499893 2576 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 00:55:14.507411 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499896 2576 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 00:55:14.507411 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499899 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 00:55:14.507411 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499902 2576 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 00:55:14.507411 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499905 2576 flags.go:64] FLAG: --v="2" Oct 31 00:55:14.507411 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499910 2576 flags.go:64] FLAG: --version="false" Oct 31 00:55:14.507411 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499914 2576 flags.go:64] FLAG: --vmodule="" Oct 31 00:55:14.507411 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499918 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 00:55:14.507411 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.499922 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 00:55:14.507411 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500053 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:55:14.507411 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500057 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:55:14.507411 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500060 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:55:14.507411 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500064 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:55:14.507411 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500070 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:55:14.507411 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500074 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:55:14.507411 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500077 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:55:14.507411 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500080 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:55:14.507411 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500083 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:55:14.507411 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500086 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:55:14.507411 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500088 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:55:14.508247 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500091 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:55:14.508247 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500094 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:55:14.508247 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500097 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:55:14.508247 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500099 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:55:14.508247 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500102 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:55:14.508247 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500104 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:55:14.508247 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500107 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:55:14.508247 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500109 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:55:14.508247 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500113 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:55:14.508247 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500115 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:55:14.508247 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500118 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:55:14.508247 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500121 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:55:14.508247 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500123 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:55:14.508247 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500126 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:55:14.508247 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500128 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:55:14.508247 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500131 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:55:14.508247 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500133 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:55:14.508247 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500136 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:55:14.508247 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500138 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:55:14.508247 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500140 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:55:14.508765 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500143 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:55:14.508765 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500146 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:55:14.508765 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500148 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:55:14.508765 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500150 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:55:14.508765 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500153 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:55:14.508765 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500157 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:55:14.508765 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500159 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:55:14.508765 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500162 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:55:14.508765 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500164 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:55:14.508765 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500167 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:55:14.508765 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500169 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:55:14.508765 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500172 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:55:14.508765 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500174 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:55:14.508765 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500177 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:55:14.508765 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500180 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:55:14.508765 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500182 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:55:14.508765 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500185 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:55:14.508765 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500187 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:55:14.508765 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500190 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:55:14.508765 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500192 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:55:14.509252 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500194 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:55:14.509252 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500197 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:55:14.509252 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500200 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:55:14.509252 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500202 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:55:14.509252 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500205 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:55:14.509252 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500207 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:55:14.509252 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500210 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:55:14.509252 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500212 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:55:14.509252 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500215 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:55:14.509252 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500217 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:55:14.509252 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500220 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:55:14.509252 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500223 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:55:14.509252 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500226 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:55:14.509252 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500228 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:55:14.509252 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500231 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:55:14.509252 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500233 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:55:14.509252 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500236 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:55:14.509252 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500240 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:55:14.509252 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500242 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:55:14.509252 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500245 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:55:14.509783 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500247 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:55:14.509783 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500250 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:55:14.509783 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500253 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:55:14.509783 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500255 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:55:14.509783 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500257 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:55:14.509783 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500260 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:55:14.509783 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500262 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:55:14.509783 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500265 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:55:14.509783 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500268 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:55:14.509783 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500270 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:55:14.509783 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500273 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:55:14.509783 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500275 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:55:14.509783 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500278 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:55:14.509783 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500280 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:55:14.509783 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500283 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:55:14.509783 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500285 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:55:14.509783 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500289 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:55:14.509783 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500293 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:55:14.509783 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500296 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:55:14.510222 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500299 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:55:14.510222 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500301 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:55:14.510222 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500304 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:55:14.510222 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500306 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:55:14.510222 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500309 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:55:14.510222 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500311 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:55:14.510222 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500314 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:55:14.510222 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500316 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:55:14.510222 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500319 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:55:14.510222 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500321 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:55:14.510222 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500325 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:55:14.510222 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500328 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:55:14.510222 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500330 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:55:14.510222 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500333 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:55:14.510222 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500335 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:55:14.510222 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500337 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:55:14.510615 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.500364 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:55:14.510615 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500401 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:55:14.510615 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500405 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:55:14.510615 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500408 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:55:14.510615 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500413 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:55:14.510615 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500429 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:55:14.510615 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500432 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:55:14.510615 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500436 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:55:14.510615 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500438 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:55:14.510615 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500441 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:55:14.510615 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500443 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:55:14.510615 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500446 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:55:14.510615 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500449 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:55:14.510615 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500451 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:55:14.510615 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500454 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:55:14.510615 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500457 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:55:14.510990 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500459 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:55:14.510990 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500462 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:55:14.510990 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500464 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:55:14.510990 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500467 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:55:14.510990 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500469 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:55:14.510990 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500472 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:55:14.510990 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500474 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:55:14.510990 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500477 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:55:14.510990 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500479 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:55:14.510990 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500482 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:55:14.510990 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500486 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:55:14.510990 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500488 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:55:14.510990 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500491 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:55:14.510990 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500493 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:55:14.510990 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500495 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:55:14.510990 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500498 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:55:14.510990 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500500 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:55:14.510990 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500504 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:55:14.510990 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500507 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:55:14.510990 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500510 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:55:14.511593 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500512 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:55:14.511593 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500515 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:55:14.511593 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500518 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:55:14.511593 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500521 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:55:14.511593 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500523 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:55:14.511593 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500526 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:55:14.511593 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500529 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:55:14.511593 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500531 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:55:14.511593 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500534 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:55:14.511593 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500536 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:55:14.511593 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500539 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:55:14.511593 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500541 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:55:14.511593 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500544 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:55:14.511593 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500547 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:55:14.511593 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500549 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:55:14.511593 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500552 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:55:14.511593 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500554 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:55:14.511593 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500557 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:55:14.511593 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500559 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:55:14.511593 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500562 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:55:14.512059 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500564 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:55:14.512059 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500567 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:55:14.512059 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500571 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:55:14.512059 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500574 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:55:14.512059 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500576 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:55:14.512059 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500579 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:55:14.512059 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500582 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:55:14.512059 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500584 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:55:14.512059 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500587 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:55:14.512059 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500591 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:55:14.512059 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500594 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:55:14.512059 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500597 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:55:14.512059 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500599 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:55:14.512059 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500602 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:55:14.512059 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500605 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:55:14.512059 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500608 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:55:14.512059 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500611 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:55:14.512059 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500614 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:55:14.512059 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500616 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:55:14.512516 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500619 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:55:14.512516 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500621 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:55:14.512516 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500624 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:55:14.512516 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500626 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:55:14.512516 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500629 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:55:14.512516 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500631 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:55:14.512516 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500634 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:55:14.512516 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500636 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:55:14.512516 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500639 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:55:14.512516 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500641 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:55:14.512516 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500644 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:55:14.512516 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500646 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:55:14.512516 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500649 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:55:14.512516 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500653 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:55:14.512516 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500656 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:55:14.512516 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500662 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:55:14.512516 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500665 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:55:14.512516 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500667 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:55:14.512516 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500670 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:55:14.512963 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500672 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:55:14.512963 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500674 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:55:14.512963 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500677 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:55:14.512963 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500681 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:55:14.512963 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500683 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:55:14.512963 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500686 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:55:14.512963 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500688 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:55:14.512963 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500691 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:55:14.512963 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500693 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:55:14.512963 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500696 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:55:14.512963 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500699 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:55:14.512963 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500701 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:55:14.512963 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.500704 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:55:14.512963 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.509611 2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 00:55:14.512963 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.509633 2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 00:55:14.512963 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.510253 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 00:55:14.512963 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.510259 2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 00:55:14.512963 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510324 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:55:14.512963 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510331 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:55:14.513430 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510334 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:55:14.513430 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510337 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:55:14.513430 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510340 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:55:14.513430 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510343 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:55:14.513430 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510346 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:55:14.513430 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510348 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:55:14.513430 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510351 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:55:14.513430 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510354 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:55:14.513430 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510356 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:55:14.513430 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510360 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:55:14.513430 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510363 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:55:14.513430 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510365 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:55:14.513430 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510368 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:55:14.513430 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510371 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:55:14.513430 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510374 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:55:14.513430 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510377 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:55:14.513430 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510379 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:55:14.513430 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510383 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:55:14.513864 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510387 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:55:14.513864 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510392 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:55:14.513864 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510395 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:55:14.513864 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510398 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:55:14.513864 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510400 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:55:14.513864 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510403 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:55:14.513864 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510406 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:55:14.513864 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510409 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:55:14.513864 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510412 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:55:14.513864 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510430 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:55:14.513864 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510435 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:55:14.513864 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510439 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:55:14.513864 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510442 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:55:14.513864 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510446 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:55:14.513864 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510449 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:55:14.513864 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510452 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:55:14.513864 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510455 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:55:14.513864 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510458 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:55:14.513864 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510460 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:55:14.514353 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510463 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:55:14.514353 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510466 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:55:14.514353 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510468 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:55:14.514353 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510472 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:55:14.514353 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510475 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:55:14.514353 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510477 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:55:14.514353 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510480 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:55:14.514353 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510483 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:55:14.514353 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510485 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:55:14.514353 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510488 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:55:14.514353 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510491 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:55:14.514353 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510494 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:55:14.514353 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510496 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:55:14.514353 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510499 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:55:14.514353 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510501 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:55:14.514353 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510504 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:55:14.514353 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510507 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:55:14.514353 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510510 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:55:14.514353 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510512 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:55:14.514353 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510515 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:55:14.514948 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510518 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:55:14.514948 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510521 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:55:14.514948 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510523 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:55:14.514948 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510526 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:55:14.514948 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510529 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:55:14.514948 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510531 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:55:14.514948 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510534 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:55:14.514948 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510537 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:55:14.514948 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510539 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:55:14.514948 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510542 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:55:14.514948 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510545 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:55:14.514948 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510547 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:55:14.514948 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510550 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:55:14.514948 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510552 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:55:14.514948 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510555 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:55:14.514948 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510557 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:55:14.514948 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510560 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:55:14.514948 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510563 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:55:14.514948 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510565 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:55:14.514948 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510568 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:55:14.515544 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510571 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:55:14.515544 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510574 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:55:14.515544 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510576 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:55:14.515544 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510579 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:55:14.515544 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510581 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:55:14.515544 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510584 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:55:14.515544 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510587 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:55:14.515544 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510590 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:55:14.515544 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510593 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:55:14.515544 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510595 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:55:14.515544 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510598 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:55:14.515544 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510600 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:55:14.515544 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510603 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:55:14.515544 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510606 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:55:14.515544 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510608 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:55:14.515544 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510611 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:55:14.515544 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510614 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:55:14.515544 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510617 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:55:14.515544 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510620 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:55:14.515544 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510622 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:55:14.516049 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510625 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:55:14.516049 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510628 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:55:14.516049 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510631 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:55:14.516049 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510633 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:55:14.516049 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510636 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:55:14.516049 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510639 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:55:14.516049 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510642 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:55:14.516049 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.510673 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:55:14.516049 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510825 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:55:14.516049 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510830 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:55:14.516049 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510833 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:55:14.516049 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510836 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:55:14.516049 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510838 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:55:14.516049 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510842 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:55:14.516049 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510844 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:55:14.516049 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510847 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:55:14.516049 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510850 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:55:14.516501 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510852 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:55:14.516501 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510855 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:55:14.516501 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510858 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:55:14.516501 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510861 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:55:14.516501 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510864 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:55:14.516501 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510867 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:55:14.516501 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510869 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:55:14.516501 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510872 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:55:14.516501 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510875 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:55:14.516501 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510878 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:55:14.516501 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510881 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:55:14.516501 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510883 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:55:14.516501 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510886 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:55:14.516501 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510889 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:55:14.516501 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510892 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:55:14.516501 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510895 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:55:14.516501 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510897 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:55:14.516501 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510900 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:55:14.516501 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510902 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:55:14.516501 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510905 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:55:14.516982 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510907 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:55:14.516982 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510910 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:55:14.516982 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510913 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:55:14.516982 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510915 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:55:14.516982 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510918 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:55:14.516982 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510920 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:55:14.516982 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510923 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:55:14.516982 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510925 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:55:14.516982 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510928 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:55:14.516982 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510931 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:55:14.516982 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510934 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:55:14.516982 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510936 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:55:14.516982 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510939 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:55:14.516982 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510941 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:55:14.516982 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510944 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:55:14.516982 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510947 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:55:14.516982 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510950 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:55:14.516982 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510952 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:55:14.516982 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510955 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:55:14.516982 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510958 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:55:14.517549 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510961 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:55:14.517549 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510963 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:55:14.517549 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510966 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:55:14.517549 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510968 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:55:14.517549 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510972 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:55:14.517549 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510976 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:55:14.517549 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510979 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:55:14.517549 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510982 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:55:14.517549 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510984 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:55:14.517549 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510987 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:55:14.517549 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510990 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:55:14.517549 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510992 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:55:14.517549 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510995 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:55:14.517549 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.510998 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:55:14.517549 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511000 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:55:14.517549 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511003 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:55:14.517549 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511006 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:55:14.517549 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511009 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:55:14.517549 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511011 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:55:14.518025 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511014 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:55:14.518025 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511018 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:55:14.518025 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511022 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:55:14.518025 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511026 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:55:14.518025 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511029 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:55:14.518025 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511031 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:55:14.518025 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511034 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:55:14.518025 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511037 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:55:14.518025 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511039 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:55:14.518025 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511042 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:55:14.518025 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511044 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:55:14.518025 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511047 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:55:14.518025 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511050 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:55:14.518025 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511053 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:55:14.518025 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511055 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:55:14.518025 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511058 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:55:14.518025 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511061 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:55:14.518025 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511064 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:55:14.518025 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511066 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:55:14.518541 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511069 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:55:14.518541 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511072 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:55:14.518541 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511074 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:55:14.518541 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511077 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:55:14.518541 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511080 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:55:14.518541 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511082 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:55:14.518541 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511085 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:55:14.518541 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511088 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:55:14.518541 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511090 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:55:14.518541 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511092 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:55:14.518541 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511095 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:55:14.518541 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511097 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:55:14.518541 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511100 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:55:14.518541 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511103 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:55:14.518541 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511105 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:55:14.518541 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511108 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:55:14.518541 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511111 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:55:14.518541 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511114 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:55:14.518541 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511116 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:55:14.519121 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.511140 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:55:14.519121 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511174 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:55:14.519121 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511178 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:55:14.519121 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511181 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:55:14.519121 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511185 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:55:14.519121 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511187 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:55:14.519121 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511191 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:55:14.519121 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511193 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:55:14.519121 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511196 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:55:14.519121 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511199 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:55:14.519121 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511201 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:55:14.519121 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511204 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:55:14.519121 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511207 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:55:14.519121 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511209 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:55:14.519121 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511212 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:55:14.519121 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511215 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:55:14.519787 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511217 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:55:14.519787 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511220 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:55:14.519787 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511223 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:55:14.519787 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511225 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:55:14.519787 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511227 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:55:14.519787 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511230 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:55:14.519787 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511233 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:55:14.519787 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511235 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:55:14.519787 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511238 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:55:14.519787 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511241 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:55:14.519787 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511245 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:55:14.519787 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511249 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:55:14.519787 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511252 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:55:14.519787 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511254 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:55:14.519787 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511257 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:55:14.519787 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511260 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:55:14.519787 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511263 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:55:14.519787 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511265 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:55:14.519787 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511268 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:55:14.520326 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511270 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:55:14.520326 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511273 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:55:14.520326 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511275 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:55:14.520326 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511278 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:55:14.520326 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511281 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:55:14.520326 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511283 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:55:14.520326 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511287 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:55:14.520326 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511291 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:55:14.520326 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511294 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:55:14.520326 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511296 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:55:14.520326 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511299 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:55:14.520326 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511302 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:55:14.520326 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511304 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:55:14.520326 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511307 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:55:14.520326 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511310 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:55:14.520326 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511312 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:55:14.520326 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511315 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:55:14.520326 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511317 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:55:14.520326 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511320 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:55:14.520958 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511322 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:55:14.520958 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511325 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:55:14.520958 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511328 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:55:14.520958 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511330 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:55:14.520958 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511333 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:55:14.520958 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511335 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:55:14.520958 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511338 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:55:14.520958 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511340 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:55:14.520958 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511344 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:55:14.520958 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511347 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:55:14.520958 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511350 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:55:14.520958 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511353 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:55:14.520958 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511356 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:55:14.520958 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511358 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:55:14.520958 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511361 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:55:14.520958 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511363 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:55:14.520958 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511366 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:55:14.520958 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511368 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:55:14.520958 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511371 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:55:14.520958 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511373 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:55:14.521723 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511376 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:55:14.521723 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511378 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:55:14.521723 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511381 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:55:14.521723 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511383 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:55:14.521723 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511386 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:55:14.521723 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511389 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:55:14.521723 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511391 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:55:14.521723 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511394 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:55:14.521723 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511396 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:55:14.521723 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511399 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:55:14.521723 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511401 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:55:14.521723 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511404 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:55:14.521723 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511409 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:55:14.521723 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511412 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:55:14.521723 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511428 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:55:14.521723 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511432 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:55:14.521723 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511435 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:55:14.521723 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511437 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:55:14.521723 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511440 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:55:14.521723 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511443 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:55:14.522267 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511446 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:55:14.522267 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511449 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:55:14.522267 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511452 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:55:14.522267 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511455 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:55:14.522267 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511458 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:55:14.522267 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511461 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:55:14.522267 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511463 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:55:14.522267 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511468 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:55:14.522267 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511470 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:55:14.522267 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511473 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:55:14.522267 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511475 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:55:14.522267 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511478 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:55:14.522267 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:14.511481 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:55:14.522267 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.511562 2576 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 00:55:14.522267 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.513669 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 00:55:14.522267 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.514639 2576 server.go:1014] "Starting client certificate rotation" Oct 31 00:55:14.522267 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.514749 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 00:55:14.522267 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.515492 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 00:55:14.522766 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.516549 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:55:14.522766 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.517748 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:55:14.527687 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.527667 2576 log.go:25] "Validated CRI v1 runtime API" Oct 31 00:55:14.533412 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.533391 2576 log.go:25] "Validated CRI v1 image API" Oct 31 00:55:14.534412 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.534370 2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 00:55:14.537172 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.537150 2576 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d8d4d7a8-786a-4c78-b06d-075d767b5b21:/dev/nvme0n1p3 f7d2d5b4-2150-4078-ac14-7df7b7fc6241:/dev/nvme0n1p4] Oct 31 00:55:14.537248 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.537173 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 00:55:14.544067 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.543932 2576 manager.go:217] Machine: {Timestamp:2026-10-31 00:55:14.54259662 +0000 UTC m=+0.357795484 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101086 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec223bd617052241626d969ccd19a66a SystemUUID:ec223bd6-1705-2241-626d-969ccd19a66a BootID:3df2f492-315d-4014-9f17-4520f8fd14f3 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:da:8f:44:23 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:da:8f:44:23 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:aa:13:c6:01:01:4d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 00:55:14.544632 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.544619 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 00:55:14.544742 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.544729 2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 00:55:14.546374 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.546354 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 00:55:14.547444 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.547396 2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 00:55:14.547591 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.547444 2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-39.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 00:55:14.547643 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.547594 2576 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 00:55:14.547643 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.547603 2576 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 00:55:14.547701 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.547643 2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:55:14.547701 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.547663 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:55:14.548513 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.548502 2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 00:55:14.552487 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.552471 2576 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 00:55:14.552634 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.552624 2576 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 00:55:14.552710 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.552701 2576 kubelet.go:493] "Attempting to sync node with API server" Oct 31 00:55:14.552746 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.552715 2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 00:55:14.552746 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.552728 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 00:55:14.552746 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.552741 2576 kubelet.go:405] "Adding apiserver pod source" Oct 31 00:55:14.552872 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.552752 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 00:55:14.553238 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.553223 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:55:14.556118 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.556100 2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 00:55:14.562066 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.562047 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:55:14.562845 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.562832 2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 00:55:14.562885 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.562865 2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 00:55:14.563044 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.563034 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 00:55:14.563080 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.563050 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 00:55:14.563080 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.563056 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 00:55:14.563080 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.563062 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 00:55:14.563080 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.563067 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 00:55:14.563080 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.563073 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 00:55:14.563664 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.563655 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 00:55:14.563704 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.563666 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 00:55:14.563704 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.563673 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 00:55:14.563704 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.563680 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 00:55:14.563704 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.563695 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 00:55:14.563704 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.563703 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 00:55:14.564279 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.564267 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 00:55:14.564279 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.564279 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 00:55:14.568413 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.568398 2576 server.go:1263] "Started kubelet" Oct 31 00:55:14.568528 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.568481 2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 00:55:14.568567 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.568515 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 00:55:14.568595 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.568586 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 00:55:14.569452 ip-10-0-130-39 systemd[1]: Started Kubernetes Kubelet. Oct 31 00:55:14.571192 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.571048 2576 server.go:317] "Adding debug handlers to kubelet server" Oct 31 00:55:14.571444 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.571404 2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 00:55:14.576691 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.576665 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-39.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 31 00:55:14.576779 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:14.576688 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-39.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 31 00:55:14.576779 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:14.576684 2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 00:55:14.576881 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:14.576857 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 31 00:55:14.577232 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.577220 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 00:55:14.577729 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:14.576690 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-39.us-east-2.compute.internal.18e3772da6268fd7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-39.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-39.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 00:55:14.568368087 +0000 UTC m=+0.383566950,LastTimestamp:2026-10-31 00:55:14.568368087 +0000 UTC m=+0.383566950,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-39.us-east-2.compute.internal,}" Oct 31 00:55:14.578300 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.578284 2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 00:55:14.578443 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:14.578410 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-39.us-east-2.compute.internal\" not found" Oct 31 00:55:14.579060 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.579044 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 00:55:14.579060 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.579046 2576 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 00:55:14.579223 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.579066 2576 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 00:55:14.579223 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.579101 2576 factory.go:55] Registering systemd factory Oct 31 00:55:14.579223 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.579116 2576 factory.go:223] Registration of the systemd container factory successfully Oct 31 00:55:14.579223 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.579118 2576 reconstruct.go:97] "Volume reconstruction finished" Oct 31 00:55:14.579223 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.579172 2576 reconciler.go:29] "Reconciler: start to sync state" Oct 31 00:55:14.579504 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.579438 2576 factory.go:153] Registering CRI-O factory Oct 31 00:55:14.579504 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.579453 2576 factory.go:223] Registration of the crio container factory successfully Oct 31 00:55:14.579565 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.579507 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 00:55:14.579565 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.579526 2576 factory.go:103] Registering Raw factory Oct 31 00:55:14.579565 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.579537 2576 manager.go:1199] Started watching for new ooms in manager Oct 31 00:55:14.580616 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.580600 2576 manager.go:319] Starting recovery of all containers Oct 31 00:55:14.580871 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.580843 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ljl7x" Oct 31 00:55:14.581034 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:14.581013 2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-39.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 31 00:55:14.588313 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:14.588285 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 31 00:55:14.591009 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.590963 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ljl7x" Oct 31 00:55:14.591533 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.591517 2576 manager.go:324] Recovery completed Oct 31 00:55:14.596377 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.596361 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:55:14.602869 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.602847 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-39.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:55:14.602987 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.602889 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-39.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:55:14.602987 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.602905 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-39.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:55:14.603499 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.603486 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:55:14.603587 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.603486 2576 cpu_manager.go:226] "Starting" policy="none" Oct 31 00:55:14.603587 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.603541 2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 00:55:14.603587 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.603563 2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 00:55:14.604691 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.604674 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-39.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:55:14.604751 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.604706 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-39.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:55:14.604751 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.604721 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-39.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:55:14.605410 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:14.605324 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-39.us-east-2.compute.internal.18e3772da8350b4d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-39.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-39.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-39.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 00:55:14.602871629 +0000 UTC m=+0.418070504,LastTimestamp:2026-10-31 00:55:14.602871629 +0000 UTC m=+0.418070504,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-39.us-east-2.compute.internal,}" Oct 31 00:55:14.605592 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.605579 2576 policy_none.go:50] "Start" Oct 31 00:55:14.605656 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.605598 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 00:55:14.605656 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.605613 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 00:55:14.607543 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.607529 2576 policy_none.go:44] "Start" Oct 31 00:55:14.660378 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.646151 2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 00:55:14.660378 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.647215 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 00:55:14.660378 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.648565 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 00:55:14.660378 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.648596 2576 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 00:55:14.660378 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.648615 2576 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 00:55:14.660378 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:14.648649 2576 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 00:55:14.660378 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.651524 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:55:14.660378 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.652729 2576 manager.go:341] "Starting Device Plugin manager" Oct 31 00:55:14.660378 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:14.652754 2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 00:55:14.660378 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.652763 2576 server.go:87] "Starting device plugin registration server" Oct 31 00:55:14.660378 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.653011 2576 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 00:55:14.660378 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.653022 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 00:55:14.660378 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.653135 2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 00:55:14.660378 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.653214 2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 00:55:14.660378 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.653222 2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 00:55:14.660378 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:14.653782 2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 00:55:14.660378 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:14.653830 2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-39.us-east-2.compute.internal\" not found" Oct 31 00:55:14.749823 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.749731 2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-39.us-east-2.compute.internal"] Oct 31 00:55:14.749823 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.749821 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:55:14.750867 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.750850 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-39.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:55:14.750975 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.750879 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-39.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:55:14.750975 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.750891 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-39.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:55:14.752655 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.752643 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:55:14.752809 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.752794 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal" Oct 31 00:55:14.752865 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.752835 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:55:14.753147 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.753131 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:55:14.753509 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.753485 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-39.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:55:14.753509 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.753509 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-39.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:55:14.753605 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.753519 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-39.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:55:14.753605 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.753554 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-39.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:55:14.753605 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.753584 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-39.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:55:14.753605 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.753601 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-39.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:55:14.753806 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.753793 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-39.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:55:14.753849 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.753815 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-39.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:55:14.753849 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.753823 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-39.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:55:14.753849 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.753840 2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-39.us-east-2.compute.internal" Oct 31 00:55:14.755307 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.755294 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-39.us-east-2.compute.internal" Oct 31 00:55:14.755354 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.755318 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:55:14.756312 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.756054 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-39.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:55:14.756312 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.756083 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-39.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:55:14.756312 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.756097 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-39.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:55:14.761696 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.761682 2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-39.us-east-2.compute.internal" Oct 31 00:55:14.761763 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:14.761704 2576 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-39.us-east-2.compute.internal\": node \"ip-10-0-130-39.us-east-2.compute.internal\" not found" Oct 31 00:55:14.777789 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:14.777615 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-39.us-east-2.compute.internal\" not found" Oct 31 00:55:14.784601 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:14.784580 2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-39.us-east-2.compute.internal\" not found" node="ip-10-0-130-39.us-east-2.compute.internal" Oct 31 00:55:14.788318 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:14.788297 2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-39.us-east-2.compute.internal\" not found" node="ip-10-0-130-39.us-east-2.compute.internal" Oct 31 00:55:14.878331 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:14.878294 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-39.us-east-2.compute.internal\" not found" Oct 31 00:55:14.880521 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.880499 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ce6a902de7718819f66be8130076b8e6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal\" (UID: \"ce6a902de7718819f66be8130076b8e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal" Oct 31 00:55:14.880601 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.880529 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce6a902de7718819f66be8130076b8e6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal\" (UID: \"ce6a902de7718819f66be8130076b8e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal" Oct 31 00:55:14.880601 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.880551 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eaafb8f2be56642836cc3cf93864aa97-config\") pod \"kube-apiserver-proxy-ip-10-0-130-39.us-east-2.compute.internal\" (UID: \"eaafb8f2be56642836cc3cf93864aa97\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-39.us-east-2.compute.internal" Oct 31 00:55:14.978477 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:14.978401 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-39.us-east-2.compute.internal\" not found" Oct 31 00:55:14.981691 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.981672 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ce6a902de7718819f66be8130076b8e6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal\" (UID: \"ce6a902de7718819f66be8130076b8e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal" Oct 31 00:55:14.981747 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.981700 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce6a902de7718819f66be8130076b8e6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal\" (UID: \"ce6a902de7718819f66be8130076b8e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal" Oct 31 00:55:14.981747 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.981717 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eaafb8f2be56642836cc3cf93864aa97-config\") pod \"kube-apiserver-proxy-ip-10-0-130-39.us-east-2.compute.internal\" (UID: \"eaafb8f2be56642836cc3cf93864aa97\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-39.us-east-2.compute.internal" Oct 31 00:55:14.981808 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.981770 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eaafb8f2be56642836cc3cf93864aa97-config\") pod \"kube-apiserver-proxy-ip-10-0-130-39.us-east-2.compute.internal\" (UID: \"eaafb8f2be56642836cc3cf93864aa97\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-39.us-east-2.compute.internal" Oct 31 00:55:14.981808 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.981775 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ce6a902de7718819f66be8130076b8e6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal\" (UID: \"ce6a902de7718819f66be8130076b8e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal" Oct 31 00:55:14.981808 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:14.981786 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce6a902de7718819f66be8130076b8e6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal\" (UID: \"ce6a902de7718819f66be8130076b8e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal" Oct 31 00:55:15.079200 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:15.079109 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-39.us-east-2.compute.internal\" not found" Oct 31 00:55:15.088482 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:15.088461 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal" Oct 31 00:55:15.091511 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:15.091491 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-39.us-east-2.compute.internal" Oct 31 00:55:15.179262 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:15.179213 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-39.us-east-2.compute.internal\" not found" Oct 31 00:55:15.279664 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:15.279626 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-39.us-east-2.compute.internal\" not found" Oct 31 00:55:15.380039 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:15.380002 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-39.us-east-2.compute.internal\" not found" Oct 31 00:55:15.480992 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:15.480963 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-39.us-east-2.compute.internal\" not found" Oct 31 00:55:15.514819 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:15.514787 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 00:55:15.515316 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:15.514938 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 00:55:15.581202 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:15.581168 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-39.us-east-2.compute.internal\" not found" Oct 31 00:55:15.595945 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:15.595882 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 00:50:15 +0000 UTC" deadline="2028-05-04 13:16:37.192652648 +0000 UTC" Oct 31 00:55:15.595945 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:15.595939 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13236h21m21.596716165s" Oct 31 00:55:15.621256 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:15.621193 2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:55:15.681566 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:15.681538 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-39.us-east-2.compute.internal\" not found" Oct 31 00:55:15.782228 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:15.782204 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-39.us-east-2.compute.internal\" not found" Oct 31 00:55:15.789531 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:15.789497 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaafb8f2be56642836cc3cf93864aa97.slice/crio-cd31ba38051bd98153d6964a968a80f77392c88ca1a73ad450f84bf749ba8b49 WatchSource:0}: Error finding container cd31ba38051bd98153d6964a968a80f77392c88ca1a73ad450f84bf749ba8b49: Status 404 returned error can't find the container with id cd31ba38051bd98153d6964a968a80f77392c88ca1a73ad450f84bf749ba8b49 Oct 31 00:55:15.794444 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:15.794430 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:55:15.799616 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:15.799594 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce6a902de7718819f66be8130076b8e6.slice/crio-d80b2fdc62f60d2d25cc7c426407eec80ec1fedfb57f6e806624299d38dc9a14 WatchSource:0}: Error finding container d80b2fdc62f60d2d25cc7c426407eec80ec1fedfb57f6e806624299d38dc9a14: Status 404 returned error can't find the container with id d80b2fdc62f60d2d25cc7c426407eec80ec1fedfb57f6e806624299d38dc9a14 Oct 31 00:55:15.883190 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:15.883104 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-39.us-east-2.compute.internal\" not found" Oct 31 00:55:15.910987 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:15.910961 2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:55:15.983678 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:15.983645 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-39.us-east-2.compute.internal\" not found" Oct 31 00:55:16.083865 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:16.083828 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-39.us-east-2.compute.internal\" not found" Oct 31 00:55:16.117694 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.117670 2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:55:16.178841 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.178776 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal" Oct 31 00:55:16.190992 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.190972 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:55:16.192106 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.192093 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-39.us-east-2.compute.internal" Oct 31 00:55:16.201026 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.201007 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:55:16.554626 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.554553 2576 apiserver.go:52] "Watching apiserver" Oct 31 00:55:16.565649 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.565613 2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 00:55:16.566023 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.566003 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-5xvng","openshift-network-operator/iptables-alerter-mtwcc","openshift-ovn-kubernetes/ovnkube-node-vcdmb","kube-system/global-pull-secret-syncer-s9tlh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg","openshift-dns/node-resolver-qsppl","openshift-multus/multus-additional-cni-plugins-p8vd6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal","openshift-multus/network-metrics-daemon-pcvbh","kube-system/konnectivity-agent-8kln5","kube-system/kube-apiserver-proxy-ip-10-0-130-39.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-8r7qf","openshift-image-registry/node-ca-qzm2t","openshift-multus/multus-g5h4s"] Oct 31 00:55:16.568794 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.568771 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-p8vd6" Oct 31 00:55:16.571229 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.571203 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qzm2t" Oct 31 00:55:16.573120 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.573093 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 00:55:16.573120 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.573093 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 00:55:16.573295 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.573132 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 00:55:16.573359 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.573313 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 00:55:16.573558 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.573464 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-m2wnh\"" Oct 31 00:55:16.573558 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.573534 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 00:55:16.574034 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.574018 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.574339 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.574258 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 00:55:16.574803 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.574780 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 00:55:16.574906 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.574792 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 00:55:16.575054 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.575037 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4j5v2\"" Oct 31 00:55:16.576806 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.576786 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s9tlh" Oct 31 00:55:16.576937 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:16.576909 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s9tlh" podUID="49f8a438-6b29-4160-98e1-3f26e27fe0af" Oct 31 00:55:16.577115 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.577022 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xcm4p\"" Oct 31 00:55:16.578576 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.577247 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 00:55:16.578576 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.577632 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 00:55:16.578576 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.577687 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 00:55:16.578576 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.577753 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 00:55:16.578576 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.577802 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 00:55:16.578576 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.578548 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 00:55:16.579095 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.578794 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 00:55:16.579824 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.579804 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:16.582549 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.582525 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qsppl" Oct 31 00:55:16.585578 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.585559 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pcvbh" Oct 31 00:55:16.585676 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:16.585653 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pcvbh" podUID="0fe8107c-9128-4c8a-9bde-e1cacd2ed19f" Oct 31 00:55:16.588333 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.588314 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xvng" Oct 31 00:55:16.588444 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:16.588394 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xvng" podUID="3c9845ea-a9d6-4d5c-93ea-fb096666a2b9" Oct 31 00:55:16.591079 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.591051 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-systemd-units\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.591191 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.591094 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-host-cni-bin\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.591191 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.591122 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-ovn-node-metrics-cert\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.591191 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.591151 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88c1a03a-cb94-4226-9d53-cd923895569e-system-cni-dir\") pod \"multus-additional-cni-plugins-p8vd6\" (UID: \"88c1a03a-cb94-4226-9d53-cd923895569e\") " pod="openshift-multus/multus-additional-cni-plugins-p8vd6" Oct 31 00:55:16.591191 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.591178 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/88c1a03a-cb94-4226-9d53-cd923895569e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p8vd6\" (UID: \"88c1a03a-cb94-4226-9d53-cd923895569e\") " pod="openshift-multus/multus-additional-cni-plugins-p8vd6" Oct 31 00:55:16.591414 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.591206 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-986bh\" (UniqueName: \"kubernetes.io/projected/88c1a03a-cb94-4226-9d53-cd923895569e-kube-api-access-986bh\") pod \"multus-additional-cni-plugins-p8vd6\" (UID: \"88c1a03a-cb94-4226-9d53-cd923895569e\") " pod="openshift-multus/multus-additional-cni-plugins-p8vd6" Oct 31 00:55:16.591414 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.591232 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-log-socket\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.591414 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.591259 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-env-overrides\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.591414 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.591337 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-var-lib-openvswitch\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.591414 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.591363 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-host-kubelet\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.591691 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.591443 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-node-log\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.591691 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.591522 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-host-cni-netd\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.591691 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.591561 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.591691 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.591593 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mtwcc" Oct 31 00:55:16.591892 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.591606 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/49f8a438-6b29-4160-98e1-3f26e27fe0af-original-pull-secret\") pod \"global-pull-secret-syncer-s9tlh\" (UID: \"49f8a438-6b29-4160-98e1-3f26e27fe0af\") " pod="kube-system/global-pull-secret-syncer-s9tlh" Oct 31 00:55:16.592037 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.591901 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c3939885-f5d4-4bd2-a69f-435c2a8fa048-registration-dir\") pod \"aws-ebs-csi-driver-node-cq2wg\" (UID: \"c3939885-f5d4-4bd2-a69f-435c2a8fa048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:16.592037 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.591929 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-ovnkube-config\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.592037 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.591953 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c3939885-f5d4-4bd2-a69f-435c2a8fa048-sys-fs\") pod \"aws-ebs-csi-driver-node-cq2wg\" (UID: \"c3939885-f5d4-4bd2-a69f-435c2a8fa048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:16.592037 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.591971 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/88c1a03a-cb94-4226-9d53-cd923895569e-cnibin\") pod \"multus-additional-cni-plugins-p8vd6\" (UID: \"88c1a03a-cb94-4226-9d53-cd923895569e\") " pod="openshift-multus/multus-additional-cni-plugins-p8vd6" Oct 31 00:55:16.592037 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.592007 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/88c1a03a-cb94-4226-9d53-cd923895569e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p8vd6\" (UID: \"88c1a03a-cb94-4226-9d53-cd923895569e\") " pod="openshift-multus/multus-additional-cni-plugins-p8vd6" Oct 31 00:55:16.592037 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.592024 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/88c1a03a-cb94-4226-9d53-cd923895569e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p8vd6\" (UID: \"88c1a03a-cb94-4226-9d53-cd923895569e\") " pod="openshift-multus/multus-additional-cni-plugins-p8vd6" Oct 31 00:55:16.592037 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.592039 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-host-run-netns\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.592409 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.592054 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-run-systemd\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.592409 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.592079 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-host-run-ovn-kubernetes\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.592409 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.592094 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c3939885-f5d4-4bd2-a69f-435c2a8fa048-etc-selinux\") pod \"aws-ebs-csi-driver-node-cq2wg\" (UID: \"c3939885-f5d4-4bd2-a69f-435c2a8fa048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:16.592409 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.592109 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/49f8a438-6b29-4160-98e1-3f26e27fe0af-kubelet-config\") pod \"global-pull-secret-syncer-s9tlh\" (UID: \"49f8a438-6b29-4160-98e1-3f26e27fe0af\") " pod="kube-system/global-pull-secret-syncer-s9tlh" Oct 31 00:55:16.592409 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.592129 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0a37f05b-0755-4ca7-8d9b-12030234bec1-serviceca\") pod \"node-ca-qzm2t\" (UID: \"0a37f05b-0755-4ca7-8d9b-12030234bec1\") " pod="openshift-image-registry/node-ca-qzm2t" Oct 31 00:55:16.592409 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.592152 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-run-openvswitch\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.592409 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.592174 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-run-ovn\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.592409 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.592299 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-ovnkube-script-lib\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.592409 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.592335 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-host-slash\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.592409 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.592366 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdbsq\" (UniqueName: \"kubernetes.io/projected/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-kube-api-access-fdbsq\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.592409 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.592403 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c3939885-f5d4-4bd2-a69f-435c2a8fa048-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cq2wg\" (UID: \"c3939885-f5d4-4bd2-a69f-435c2a8fa048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:16.593065 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.592459 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j6vc\" (UniqueName: \"kubernetes.io/projected/c3939885-f5d4-4bd2-a69f-435c2a8fa048-kube-api-access-4j6vc\") pod \"aws-ebs-csi-driver-node-cq2wg\" (UID: \"c3939885-f5d4-4bd2-a69f-435c2a8fa048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:16.593065 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.592524 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/88c1a03a-cb94-4226-9d53-cd923895569e-cni-binary-copy\") pod \"multus-additional-cni-plugins-p8vd6\" (UID: \"88c1a03a-cb94-4226-9d53-cd923895569e\") " pod="openshift-multus/multus-additional-cni-plugins-p8vd6" Oct 31 00:55:16.593065 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.592559 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-etc-openvswitch\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.593065 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.592593 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/49f8a438-6b29-4160-98e1-3f26e27fe0af-dbus\") pod \"global-pull-secret-syncer-s9tlh\" (UID: \"49f8a438-6b29-4160-98e1-3f26e27fe0af\") " pod="kube-system/global-pull-secret-syncer-s9tlh" Oct 31 00:55:16.593065 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.592621 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c3939885-f5d4-4bd2-a69f-435c2a8fa048-socket-dir\") pod \"aws-ebs-csi-driver-node-cq2wg\" (UID: \"c3939885-f5d4-4bd2-a69f-435c2a8fa048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:16.593065 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.592648 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c3939885-f5d4-4bd2-a69f-435c2a8fa048-device-dir\") pod \"aws-ebs-csi-driver-node-cq2wg\" (UID: \"c3939885-f5d4-4bd2-a69f-435c2a8fa048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:16.593065 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.592672 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a37f05b-0755-4ca7-8d9b-12030234bec1-host\") pod \"node-ca-qzm2t\" (UID: \"0a37f05b-0755-4ca7-8d9b-12030234bec1\") " pod="openshift-image-registry/node-ca-qzm2t" Oct 31 00:55:16.593065 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.592723 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npk86\" (UniqueName: \"kubernetes.io/projected/0a37f05b-0755-4ca7-8d9b-12030234bec1-kube-api-access-npk86\") pod \"node-ca-qzm2t\" (UID: \"0a37f05b-0755-4ca7-8d9b-12030234bec1\") " pod="openshift-image-registry/node-ca-qzm2t" Oct 31 00:55:16.595308 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.595291 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8kln5" Oct 31 00:55:16.598055 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.598035 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.601224 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.601188 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.655099 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.655044 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal" event={"ID":"ce6a902de7718819f66be8130076b8e6","Type":"ContainerStarted","Data":"d80b2fdc62f60d2d25cc7c426407eec80ec1fedfb57f6e806624299d38dc9a14"} Oct 31 00:55:16.656638 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.656608 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-39.us-east-2.compute.internal" event={"ID":"eaafb8f2be56642836cc3cf93864aa97","Type":"ContainerStarted","Data":"cd31ba38051bd98153d6964a968a80f77392c88ca1a73ad450f84bf749ba8b49"} Oct 31 00:55:16.680601 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.680573 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 00:55:16.681914 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.681888 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 00:55:16.681914 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.681905 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 00:55:16.682100 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.682000 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 00:55:16.682100 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.682075 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 00:55:16.682100 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.682078 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 00:55:16.682250 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.682118 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 00:55:16.682250 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.682148 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 00:55:16.682250 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.682166 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9dpj5\"" Oct 31 00:55:16.682250 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.682184 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 00:55:16.682250 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.682193 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-jrg45\"" Oct 31 00:55:16.682250 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.682202 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:55:16.682250 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.682211 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 00:55:16.682749 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.682734 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:55:16.683111 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.683088 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-j9qsc\"" Oct 31 00:55:16.683163 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.683130 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 00:55:16.683163 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.683141 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-xxmpd\"" Oct 31 00:55:16.683238 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.683142 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-dlpqk\"" Oct 31 00:55:16.683238 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.683203 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 00:55:16.683317 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.683264 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-89rf9\"" Oct 31 00:55:16.692900 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.692876 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-run\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.693023 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.692920 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c3939885-f5d4-4bd2-a69f-435c2a8fa048-etc-selinux\") pod \"aws-ebs-csi-driver-node-cq2wg\" (UID: \"c3939885-f5d4-4bd2-a69f-435c2a8fa048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:16.693023 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.692952 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d610021a-d4d9-473d-8889-cb5afcb5b8a0-cni-binary-copy\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.693023 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.692978 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4b8ba0db-d8b9-4a40-a144-1400634cd99d-iptables-alerter-script\") pod \"iptables-alerter-mtwcc\" (UID: \"4b8ba0db-d8b9-4a40-a144-1400634cd99d\") " pod="openshift-network-operator/iptables-alerter-mtwcc" Oct 31 00:55:16.693023 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.693007 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-etc-kubernetes\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.693238 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.693201 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c3939885-f5d4-4bd2-a69f-435c2a8fa048-etc-selinux\") pod \"aws-ebs-csi-driver-node-cq2wg\" (UID: \"c3939885-f5d4-4bd2-a69f-435c2a8fa048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:16.693286 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.693237 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0a37f05b-0755-4ca7-8d9b-12030234bec1-serviceca\") pod \"node-ca-qzm2t\" (UID: \"0a37f05b-0755-4ca7-8d9b-12030234bec1\") " pod="openshift-image-registry/node-ca-qzm2t" Oct 31 00:55:16.693330 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.693291 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-run-openvswitch\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.693330 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.693322 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-host-run-k8s-cni-cncf-io\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.693398 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.693346 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-multus-conf-dir\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.693398 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.693369 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-sys\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.693518 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.693393 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnvxk\" (UniqueName: \"kubernetes.io/projected/a6ddfc5f-887b-4821-bac9-54cb59c50e32-kube-api-access-fnvxk\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.693518 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.693422 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdbsq\" (UniqueName: \"kubernetes.io/projected/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-kube-api-access-fdbsq\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.693518 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.693448 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/12002bf6-fce5-4d20-a62c-6782522acd47-konnectivity-ca\") pod \"konnectivity-agent-8kln5\" (UID: \"12002bf6-fce5-4d20-a62c-6782522acd47\") " pod="kube-system/konnectivity-agent-8kln5" Oct 31 00:55:16.693648 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.693505 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-etc-modprobe-d\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.693648 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.693552 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-host\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.693648 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.693577 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-etc-openvswitch\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.693648 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.693600 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/49f8a438-6b29-4160-98e1-3f26e27fe0af-dbus\") pod \"global-pull-secret-syncer-s9tlh\" (UID: \"49f8a438-6b29-4160-98e1-3f26e27fe0af\") " pod="kube-system/global-pull-secret-syncer-s9tlh" Oct 31 00:55:16.693648 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.693623 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c3939885-f5d4-4bd2-a69f-435c2a8fa048-device-dir\") pod \"aws-ebs-csi-driver-node-cq2wg\" (UID: \"c3939885-f5d4-4bd2-a69f-435c2a8fa048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:16.693895 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.693657 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd654994-98a0-4d39-aa38-270b4a0d4341-tmp-dir\") pod \"node-resolver-qsppl\" (UID: \"cd654994-98a0-4d39-aa38-270b4a0d4341\") " pod="openshift-dns/node-resolver-qsppl" Oct 31 00:55:16.693895 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.693698 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-host-var-lib-kubelet\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.693895 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.693723 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-etc-systemd\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.693895 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.693839 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6ddfc5f-887b-4821-bac9-54cb59c50e32-tmp\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.694056 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.693897 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-etc-openvswitch\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.694117 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.694098 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/49f8a438-6b29-4160-98e1-3f26e27fe0af-dbus\") pod \"global-pull-secret-syncer-s9tlh\" (UID: \"49f8a438-6b29-4160-98e1-3f26e27fe0af\") " pod="kube-system/global-pull-secret-syncer-s9tlh" Oct 31 00:55:16.694155 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.694096 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c3939885-f5d4-4bd2-a69f-435c2a8fa048-device-dir\") pod \"aws-ebs-csi-driver-node-cq2wg\" (UID: \"c3939885-f5d4-4bd2-a69f-435c2a8fa048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:16.694155 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.694142 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-run-openvswitch\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.694233 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.694186 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/88c1a03a-cb94-4226-9d53-cd923895569e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p8vd6\" (UID: \"88c1a03a-cb94-4226-9d53-cd923895569e\") " pod="openshift-multus/multus-additional-cni-plugins-p8vd6" Oct 31 00:55:16.694233 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.694202 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0a37f05b-0755-4ca7-8d9b-12030234bec1-serviceca\") pod \"node-ca-qzm2t\" (UID: \"0a37f05b-0755-4ca7-8d9b-12030234bec1\") " pod="openshift-image-registry/node-ca-qzm2t" Oct 31 00:55:16.694233 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.694223 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwrz5\" (UniqueName: \"kubernetes.io/projected/3c9845ea-a9d6-4d5c-93ea-fb096666a2b9-kube-api-access-kwrz5\") pod \"network-check-target-5xvng\" (UID: \"3c9845ea-a9d6-4d5c-93ea-fb096666a2b9\") " pod="openshift-network-diagnostics/network-check-target-5xvng" Oct 31 00:55:16.694383 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.694254 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b8ba0db-d8b9-4a40-a144-1400634cd99d-host-slash\") pod \"iptables-alerter-mtwcc\" (UID: \"4b8ba0db-d8b9-4a40-a144-1400634cd99d\") " pod="openshift-network-operator/iptables-alerter-mtwcc" Oct 31 00:55:16.694383 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.694344 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-env-overrides\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.694488 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.694388 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-multus-socket-dir-parent\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.694488 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.694402 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/88c1a03a-cb94-4226-9d53-cd923895569e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p8vd6\" (UID: \"88c1a03a-cb94-4226-9d53-cd923895569e\") " pod="openshift-multus/multus-additional-cni-plugins-p8vd6" Oct 31 00:55:16.694621 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.694585 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-var-lib-openvswitch\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.694621 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.694606 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-host-kubelet\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.694731 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.694624 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cd654994-98a0-4d39-aa38-270b4a0d4341-hosts-file\") pod \"node-resolver-qsppl\" (UID: \"cd654994-98a0-4d39-aa38-270b4a0d4341\") " pod="openshift-dns/node-resolver-qsppl" Oct 31 00:55:16.694731 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.694643 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-host-run-multus-certs\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.694731 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.694658 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/88c1a03a-cb94-4226-9d53-cd923895569e-cnibin\") pod \"multus-additional-cni-plugins-p8vd6\" (UID: \"88c1a03a-cb94-4226-9d53-cd923895569e\") " pod="openshift-multus/multus-additional-cni-plugins-p8vd6" Oct 31 00:55:16.694731 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.694673 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/88c1a03a-cb94-4226-9d53-cd923895569e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p8vd6\" (UID: \"88c1a03a-cb94-4226-9d53-cd923895569e\") " pod="openshift-multus/multus-additional-cni-plugins-p8vd6" Oct 31 00:55:16.694731 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.694689 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/88c1a03a-cb94-4226-9d53-cd923895569e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p8vd6\" (UID: \"88c1a03a-cb94-4226-9d53-cd923895569e\") " pod="openshift-multus/multus-additional-cni-plugins-p8vd6" Oct 31 00:55:16.694731 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.694706 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-host-var-lib-cni-bin\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.694731 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.694721 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nchj\" (UniqueName: \"kubernetes.io/projected/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f-kube-api-access-7nchj\") pod \"network-metrics-daemon-pcvbh\" (UID: \"0fe8107c-9128-4c8a-9bde-e1cacd2ed19f\") " pod="openshift-multus/network-metrics-daemon-pcvbh" Oct 31 00:55:16.696003 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.694735 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a6ddfc5f-887b-4821-bac9-54cb59c50e32-etc-tuned\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.696003 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.694752 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-host-run-netns\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.696003 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.694768 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-run-systemd\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.696003 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.694783 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-host-run-ovn-kubernetes\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.696003 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.694797 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-hostroot\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.696003 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.694812 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f-metrics-certs\") pod \"network-metrics-daemon-pcvbh\" (UID: \"0fe8107c-9128-4c8a-9bde-e1cacd2ed19f\") " pod="openshift-multus/network-metrics-daemon-pcvbh" Oct 31 00:55:16.696003 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.694828 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9v2m\" (UniqueName: \"kubernetes.io/projected/4b8ba0db-d8b9-4a40-a144-1400634cd99d-kube-api-access-t9v2m\") pod \"iptables-alerter-mtwcc\" (UID: \"4b8ba0db-d8b9-4a40-a144-1400634cd99d\") " pod="openshift-network-operator/iptables-alerter-mtwcc" Oct 31 00:55:16.696003 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.694843 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-lib-modules\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.696003 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.694858 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/49f8a438-6b29-4160-98e1-3f26e27fe0af-kubelet-config\") pod \"global-pull-secret-syncer-s9tlh\" (UID: \"49f8a438-6b29-4160-98e1-3f26e27fe0af\") " pod="kube-system/global-pull-secret-syncer-s9tlh" Oct 31 00:55:16.696003 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.694874 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-run-ovn\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.696003 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.694872 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-env-overrides\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.696003 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.694890 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-ovnkube-script-lib\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.696003 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.694913 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-host-var-lib-cni-multus\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.696003 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.694961 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-host-run-netns\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.696003 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.694969 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-var-lib-openvswitch\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.696003 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.694987 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-run-systemd\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.696003 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.695027 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/49f8a438-6b29-4160-98e1-3f26e27fe0af-kubelet-config\") pod \"global-pull-secret-syncer-s9tlh\" (UID: \"49f8a438-6b29-4160-98e1-3f26e27fe0af\") " pod="kube-system/global-pull-secret-syncer-s9tlh" Oct 31 00:55:16.696724 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.695036 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-host-kubelet\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.696724 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.695073 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-run-ovn\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.696724 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.695087 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/88c1a03a-cb94-4226-9d53-cd923895569e-cnibin\") pod \"multus-additional-cni-plugins-p8vd6\" (UID: \"88c1a03a-cb94-4226-9d53-cd923895569e\") " pod="openshift-multus/multus-additional-cni-plugins-p8vd6" Oct 31 00:55:16.696724 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.695121 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-etc-sysconfig\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.696724 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.695142 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-host-slash\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.696724 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.695161 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c3939885-f5d4-4bd2-a69f-435c2a8fa048-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cq2wg\" (UID: \"c3939885-f5d4-4bd2-a69f-435c2a8fa048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:16.696724 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.695203 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j6vc\" (UniqueName: \"kubernetes.io/projected/c3939885-f5d4-4bd2-a69f-435c2a8fa048-kube-api-access-4j6vc\") pod \"aws-ebs-csi-driver-node-cq2wg\" (UID: \"c3939885-f5d4-4bd2-a69f-435c2a8fa048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:16.696724 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.695221 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/88c1a03a-cb94-4226-9d53-cd923895569e-cni-binary-copy\") pod \"multus-additional-cni-plugins-p8vd6\" (UID: \"88c1a03a-cb94-4226-9d53-cd923895569e\") " pod="openshift-multus/multus-additional-cni-plugins-p8vd6" Oct 31 00:55:16.696724 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.695252 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4qw7\" (UniqueName: \"kubernetes.io/projected/cd654994-98a0-4d39-aa38-270b4a0d4341-kube-api-access-m4qw7\") pod \"node-resolver-qsppl\" (UID: \"cd654994-98a0-4d39-aa38-270b4a0d4341\") " pod="openshift-dns/node-resolver-qsppl" Oct 31 00:55:16.696724 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.695268 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-var-lib-kubelet\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.696724 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.695286 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c3939885-f5d4-4bd2-a69f-435c2a8fa048-socket-dir\") pod \"aws-ebs-csi-driver-node-cq2wg\" (UID: \"c3939885-f5d4-4bd2-a69f-435c2a8fa048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:16.696724 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.695301 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a37f05b-0755-4ca7-8d9b-12030234bec1-host\") pod \"node-ca-qzm2t\" (UID: \"0a37f05b-0755-4ca7-8d9b-12030234bec1\") " pod="openshift-image-registry/node-ca-qzm2t" Oct 31 00:55:16.696724 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.695316 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npk86\" (UniqueName: \"kubernetes.io/projected/0a37f05b-0755-4ca7-8d9b-12030234bec1-kube-api-access-npk86\") pod \"node-ca-qzm2t\" (UID: \"0a37f05b-0755-4ca7-8d9b-12030234bec1\") " pod="openshift-image-registry/node-ca-qzm2t" Oct 31 00:55:16.696724 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.695372 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-systemd-units\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.696724 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.695390 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-host-cni-bin\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.696724 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.695405 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-ovn-node-metrics-cert\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.696724 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.695421 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88c1a03a-cb94-4226-9d53-cd923895569e-system-cni-dir\") pod \"multus-additional-cni-plugins-p8vd6\" (UID: \"88c1a03a-cb94-4226-9d53-cd923895569e\") " pod="openshift-multus/multus-additional-cni-plugins-p8vd6" Oct 31 00:55:16.697438 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.695442 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-986bh\" (UniqueName: \"kubernetes.io/projected/88c1a03a-cb94-4226-9d53-cd923895569e-kube-api-access-986bh\") pod \"multus-additional-cni-plugins-p8vd6\" (UID: \"88c1a03a-cb94-4226-9d53-cd923895569e\") " pod="openshift-multus/multus-additional-cni-plugins-p8vd6" Oct 31 00:55:16.697438 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.695488 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-cnibin\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.697438 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.695513 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-log-socket\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.697438 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.695535 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-host-run-netns\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.697438 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.695572 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-ovnkube-script-lib\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.697438 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.695591 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs6kv\" (UniqueName: \"kubernetes.io/projected/d610021a-d4d9-473d-8889-cb5afcb5b8a0-kube-api-access-xs6kv\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.697438 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.695621 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-etc-sysctl-conf\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.697438 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.695643 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-node-log\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.697438 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.695660 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-host-cni-netd\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.697438 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.695556 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/88c1a03a-cb94-4226-9d53-cd923895569e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p8vd6\" (UID: \"88c1a03a-cb94-4226-9d53-cd923895569e\") " pod="openshift-multus/multus-additional-cni-plugins-p8vd6" Oct 31 00:55:16.697438 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.695709 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c3939885-f5d4-4bd2-a69f-435c2a8fa048-socket-dir\") pod \"aws-ebs-csi-driver-node-cq2wg\" (UID: \"c3939885-f5d4-4bd2-a69f-435c2a8fa048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:16.697438 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.695621 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-host-run-ovn-kubernetes\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.697438 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.695771 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.697438 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.695800 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/49f8a438-6b29-4160-98e1-3f26e27fe0af-original-pull-secret\") pod \"global-pull-secret-syncer-s9tlh\" (UID: \"49f8a438-6b29-4160-98e1-3f26e27fe0af\") " pod="kube-system/global-pull-secret-syncer-s9tlh" Oct 31 00:55:16.697438 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.695829 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c3939885-f5d4-4bd2-a69f-435c2a8fa048-registration-dir\") pod \"aws-ebs-csi-driver-node-cq2wg\" (UID: \"c3939885-f5d4-4bd2-a69f-435c2a8fa048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:16.697438 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.695858 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-system-cni-dir\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.697438 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.695870 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/88c1a03a-cb94-4226-9d53-cd923895569e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p8vd6\" (UID: \"88c1a03a-cb94-4226-9d53-cd923895569e\") " pod="openshift-multus/multus-additional-cni-plugins-p8vd6" Oct 31 00:55:16.698053 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.695885 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/12002bf6-fce5-4d20-a62c-6782522acd47-agent-certs\") pod \"konnectivity-agent-8kln5\" (UID: \"12002bf6-fce5-4d20-a62c-6782522acd47\") " pod="kube-system/konnectivity-agent-8kln5" Oct 31 00:55:16.698053 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.695913 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-ovnkube-config\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.698053 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.695933 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c3939885-f5d4-4bd2-a69f-435c2a8fa048-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cq2wg\" (UID: \"c3939885-f5d4-4bd2-a69f-435c2a8fa048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:16.698053 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.695964 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c3939885-f5d4-4bd2-a69f-435c2a8fa048-sys-fs\") pod \"aws-ebs-csi-driver-node-cq2wg\" (UID: \"c3939885-f5d4-4bd2-a69f-435c2a8fa048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:16.698053 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.695997 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-multus-cni-dir\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.698053 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.696023 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d610021a-d4d9-473d-8889-cb5afcb5b8a0-multus-daemon-config\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.698053 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.696050 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-etc-kubernetes\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.698053 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.696076 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-etc-sysctl-d\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.698053 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.696157 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-host-cni-netd\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.698053 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.696195 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a37f05b-0755-4ca7-8d9b-12030234bec1-host\") pod \"node-ca-qzm2t\" (UID: \"0a37f05b-0755-4ca7-8d9b-12030234bec1\") " pod="openshift-image-registry/node-ca-qzm2t" Oct 31 00:55:16.698053 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.696232 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-systemd-units\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.698053 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.696268 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-host-cni-bin\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.698053 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.696371 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-host-slash\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.698053 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.696373 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-log-socket\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.698053 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.696409 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-node-log\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.698053 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.696462 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.698053 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:16.696534 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:55:16.698739 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:16.696590 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49f8a438-6b29-4160-98e1-3f26e27fe0af-original-pull-secret podName:49f8a438-6b29-4160-98e1-3f26e27fe0af nodeName:}" failed. No retries permitted until 2026-10-31 00:55:17.196565065 +0000 UTC m=+3.011763963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/49f8a438-6b29-4160-98e1-3f26e27fe0af-original-pull-secret") pod "global-pull-secret-syncer-s9tlh" (UID: "49f8a438-6b29-4160-98e1-3f26e27fe0af") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:55:16.698739 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.696583 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c3939885-f5d4-4bd2-a69f-435c2a8fa048-sys-fs\") pod \"aws-ebs-csi-driver-node-cq2wg\" (UID: \"c3939885-f5d4-4bd2-a69f-435c2a8fa048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:16.698739 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.696652 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c3939885-f5d4-4bd2-a69f-435c2a8fa048-registration-dir\") pod \"aws-ebs-csi-driver-node-cq2wg\" (UID: \"c3939885-f5d4-4bd2-a69f-435c2a8fa048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:16.698739 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.696707 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88c1a03a-cb94-4226-9d53-cd923895569e-system-cni-dir\") pod \"multus-additional-cni-plugins-p8vd6\" (UID: \"88c1a03a-cb94-4226-9d53-cd923895569e\") " pod="openshift-multus/multus-additional-cni-plugins-p8vd6" Oct 31 00:55:16.698739 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.697044 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-ovnkube-config\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.698739 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.697202 2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 00:55:16.698739 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.697347 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/88c1a03a-cb94-4226-9d53-cd923895569e-cni-binary-copy\") pod \"multus-additional-cni-plugins-p8vd6\" (UID: \"88c1a03a-cb94-4226-9d53-cd923895569e\") " pod="openshift-multus/multus-additional-cni-plugins-p8vd6" Oct 31 00:55:16.701010 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.700987 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-ovn-node-metrics-cert\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.702999 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.702977 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 00:55:16.703269 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.703247 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdbsq\" (UniqueName: \"kubernetes.io/projected/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-kube-api-access-fdbsq\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.710031 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.710011 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-lzmhh" Oct 31 00:55:16.710211 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.710188 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npk86\" (UniqueName: \"kubernetes.io/projected/0a37f05b-0755-4ca7-8d9b-12030234bec1-kube-api-access-npk86\") pod \"node-ca-qzm2t\" (UID: \"0a37f05b-0755-4ca7-8d9b-12030234bec1\") " pod="openshift-image-registry/node-ca-qzm2t" Oct 31 00:55:16.710405 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.710381 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j6vc\" (UniqueName: \"kubernetes.io/projected/c3939885-f5d4-4bd2-a69f-435c2a8fa048-kube-api-access-4j6vc\") pod \"aws-ebs-csi-driver-node-cq2wg\" (UID: \"c3939885-f5d4-4bd2-a69f-435c2a8fa048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:16.713019 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.712965 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-986bh\" (UniqueName: \"kubernetes.io/projected/88c1a03a-cb94-4226-9d53-cd923895569e-kube-api-access-986bh\") pod \"multus-additional-cni-plugins-p8vd6\" (UID: \"88c1a03a-cb94-4226-9d53-cd923895569e\") " pod="openshift-multus/multus-additional-cni-plugins-p8vd6" Oct 31 00:55:16.718142 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.718121 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lzmhh" Oct 31 00:55:16.796669 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.796628 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-cnibin\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.796848 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.796677 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-host-run-netns\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.796848 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.796705 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs6kv\" (UniqueName: \"kubernetes.io/projected/d610021a-d4d9-473d-8889-cb5afcb5b8a0-kube-api-access-xs6kv\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.796848 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.796730 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-etc-sysctl-conf\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.796848 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.796759 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-cnibin\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.796848 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.796771 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-system-cni-dir\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.796848 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.796796 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/12002bf6-fce5-4d20-a62c-6782522acd47-agent-certs\") pod \"konnectivity-agent-8kln5\" (UID: \"12002bf6-fce5-4d20-a62c-6782522acd47\") " pod="kube-system/konnectivity-agent-8kln5" Oct 31 00:55:16.796848 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.796818 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-multus-cni-dir\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.796848 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.796821 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-host-run-netns\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.796848 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.796838 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d610021a-d4d9-473d-8889-cb5afcb5b8a0-multus-daemon-config\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.797326 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.796875 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-etc-kubernetes\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.797326 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.796902 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-etc-sysctl-d\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.797326 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.796921 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-multus-cni-dir\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.797326 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.796927 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-run\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.797326 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.796943 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-etc-sysctl-conf\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.797326 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.796959 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d610021a-d4d9-473d-8889-cb5afcb5b8a0-cni-binary-copy\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.797326 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.796964 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-system-cni-dir\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.797326 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.796985 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4b8ba0db-d8b9-4a40-a144-1400634cd99d-iptables-alerter-script\") pod \"iptables-alerter-mtwcc\" (UID: \"4b8ba0db-d8b9-4a40-a144-1400634cd99d\") " pod="openshift-network-operator/iptables-alerter-mtwcc" Oct 31 00:55:16.797326 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.796973 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-run\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.797326 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.797022 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-etc-kubernetes\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.797326 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.797026 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-etc-kubernetes\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.797326 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.797072 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-etc-kubernetes\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.797326 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.797076 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-host-run-k8s-cni-cncf-io\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.797326 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.797097 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-etc-sysctl-d\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.797326 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.797103 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-multus-conf-dir\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.797326 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.797136 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-sys\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.797326 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.797159 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-host-run-k8s-cni-cncf-io\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.797326 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.797164 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnvxk\" (UniqueName: \"kubernetes.io/projected/a6ddfc5f-887b-4821-bac9-54cb59c50e32-kube-api-access-fnvxk\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.798230 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.797225 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/12002bf6-fce5-4d20-a62c-6782522acd47-konnectivity-ca\") pod \"konnectivity-agent-8kln5\" (UID: \"12002bf6-fce5-4d20-a62c-6782522acd47\") " pod="kube-system/konnectivity-agent-8kln5" Oct 31 00:55:16.798230 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.797255 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-etc-modprobe-d\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.798230 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.797280 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-host\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.798230 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.797310 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd654994-98a0-4d39-aa38-270b4a0d4341-tmp-dir\") pod \"node-resolver-qsppl\" (UID: \"cd654994-98a0-4d39-aa38-270b4a0d4341\") " pod="openshift-dns/node-resolver-qsppl" Oct 31 00:55:16.798230 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.797333 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-host-var-lib-kubelet\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.798230 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.797357 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-etc-systemd\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.798230 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.797403 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6ddfc5f-887b-4821-bac9-54cb59c50e32-tmp\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.798230 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.797431 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwrz5\" (UniqueName: \"kubernetes.io/projected/3c9845ea-a9d6-4d5c-93ea-fb096666a2b9-kube-api-access-kwrz5\") pod \"network-check-target-5xvng\" (UID: \"3c9845ea-a9d6-4d5c-93ea-fb096666a2b9\") " pod="openshift-network-diagnostics/network-check-target-5xvng" Oct 31 00:55:16.798230 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.797455 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-etc-modprobe-d\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.798230 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.797456 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b8ba0db-d8b9-4a40-a144-1400634cd99d-host-slash\") pod \"iptables-alerter-mtwcc\" (UID: \"4b8ba0db-d8b9-4a40-a144-1400634cd99d\") " pod="openshift-network-operator/iptables-alerter-mtwcc" Oct 31 00:55:16.798230 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.797180 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-sys\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.798230 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.797504 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-multus-socket-dir-parent\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.798230 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.797531 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cd654994-98a0-4d39-aa38-270b4a0d4341-hosts-file\") pod \"node-resolver-qsppl\" (UID: \"cd654994-98a0-4d39-aa38-270b4a0d4341\") " pod="openshift-dns/node-resolver-qsppl" Oct 31 00:55:16.798230 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.797558 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-host-run-multus-certs\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.798230 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.797586 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-host-var-lib-cni-bin\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.798230 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.797612 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nchj\" (UniqueName: \"kubernetes.io/projected/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f-kube-api-access-7nchj\") pod \"network-metrics-daemon-pcvbh\" (UID: \"0fe8107c-9128-4c8a-9bde-e1cacd2ed19f\") " pod="openshift-multus/network-metrics-daemon-pcvbh" Oct 31 00:55:16.798230 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.797681 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4b8ba0db-d8b9-4a40-a144-1400634cd99d-iptables-alerter-script\") pod \"iptables-alerter-mtwcc\" (UID: \"4b8ba0db-d8b9-4a40-a144-1400634cd99d\") " pod="openshift-network-operator/iptables-alerter-mtwcc" Oct 31 00:55:16.798230 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.797696 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a6ddfc5f-887b-4821-bac9-54cb59c50e32-etc-tuned\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.799156 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.797701 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d610021a-d4d9-473d-8889-cb5afcb5b8a0-cni-binary-copy\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.799156 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.797744 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-hostroot\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.799156 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.797692 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d610021a-d4d9-473d-8889-cb5afcb5b8a0-multus-daemon-config\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.799156 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.797774 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f-metrics-certs\") pod \"network-metrics-daemon-pcvbh\" (UID: \"0fe8107c-9128-4c8a-9bde-e1cacd2ed19f\") " pod="openshift-multus/network-metrics-daemon-pcvbh" Oct 31 00:55:16.799156 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.797821 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9v2m\" (UniqueName: \"kubernetes.io/projected/4b8ba0db-d8b9-4a40-a144-1400634cd99d-kube-api-access-t9v2m\") pod \"iptables-alerter-mtwcc\" (UID: \"4b8ba0db-d8b9-4a40-a144-1400634cd99d\") " pod="openshift-network-operator/iptables-alerter-mtwcc" Oct 31 00:55:16.799156 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.797851 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-lib-modules\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.799156 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.797883 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-host-var-lib-cni-multus\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.799156 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.797899 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-host\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.799156 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.797903 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-etc-sysconfig\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.799156 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.797934 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-etc-sysconfig\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.799156 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.797951 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4qw7\" (UniqueName: \"kubernetes.io/projected/cd654994-98a0-4d39-aa38-270b4a0d4341-kube-api-access-m4qw7\") pod \"node-resolver-qsppl\" (UID: \"cd654994-98a0-4d39-aa38-270b4a0d4341\") " pod="openshift-dns/node-resolver-qsppl" Oct 31 00:55:16.799156 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.797982 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-var-lib-kubelet\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.799156 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.798010 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd654994-98a0-4d39-aa38-270b4a0d4341-tmp-dir\") pod \"node-resolver-qsppl\" (UID: \"cd654994-98a0-4d39-aa38-270b4a0d4341\") " pod="openshift-dns/node-resolver-qsppl" Oct 31 00:55:16.799156 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.798022 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/12002bf6-fce5-4d20-a62c-6782522acd47-konnectivity-ca\") pod \"konnectivity-agent-8kln5\" (UID: \"12002bf6-fce5-4d20-a62c-6782522acd47\") " pod="kube-system/konnectivity-agent-8kln5" Oct 31 00:55:16.799156 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.797138 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-multus-conf-dir\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.799156 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.798107 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-var-lib-kubelet\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.799156 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.798122 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-etc-systemd\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.799156 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.798162 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-host-var-lib-kubelet\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.800057 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.798162 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-hostroot\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.800057 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.798225 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cd654994-98a0-4d39-aa38-270b4a0d4341-hosts-file\") pod \"node-resolver-qsppl\" (UID: \"cd654994-98a0-4d39-aa38-270b4a0d4341\") " pod="openshift-dns/node-resolver-qsppl" Oct 31 00:55:16.800057 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.798259 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-lib-modules\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.800057 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.798271 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b8ba0db-d8b9-4a40-a144-1400634cd99d-host-slash\") pod \"iptables-alerter-mtwcc\" (UID: \"4b8ba0db-d8b9-4a40-a144-1400634cd99d\") " pod="openshift-network-operator/iptables-alerter-mtwcc" Oct 31 00:55:16.800057 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:16.797855 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:55:16.800057 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:16.798359 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f-metrics-certs podName:0fe8107c-9128-4c8a-9bde-e1cacd2ed19f nodeName:}" failed. No retries permitted until 2026-10-31 00:55:17.298338926 +0000 UTC m=+3.113537796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f-metrics-certs") pod "network-metrics-daemon-pcvbh" (UID: "0fe8107c-9128-4c8a-9bde-e1cacd2ed19f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:55:16.800057 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.798368 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-host-var-lib-cni-bin\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.800057 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.798394 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-host-run-multus-certs\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.800057 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.798455 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-multus-socket-dir-parent\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.800057 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.798527 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-host-var-lib-cni-multus\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.800578 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.800133 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/12002bf6-fce5-4d20-a62c-6782522acd47-agent-certs\") pod \"konnectivity-agent-8kln5\" (UID: \"12002bf6-fce5-4d20-a62c-6782522acd47\") " pod="kube-system/konnectivity-agent-8kln5" Oct 31 00:55:16.800578 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.800316 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a6ddfc5f-887b-4821-bac9-54cb59c50e32-etc-tuned\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.801149 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.801116 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6ddfc5f-887b-4821-bac9-54cb59c50e32-tmp\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.805220 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.805159 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs6kv\" (UniqueName: \"kubernetes.io/projected/d610021a-d4d9-473d-8889-cb5afcb5b8a0-kube-api-access-xs6kv\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.805506 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.805463 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnvxk\" (UniqueName: \"kubernetes.io/projected/a6ddfc5f-887b-4821-bac9-54cb59c50e32-kube-api-access-fnvxk\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.805702 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:16.805685 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:55:16.805785 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:16.805705 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:55:16.805785 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:16.805717 2576 projected.go:196] Error preparing data for projected volume kube-api-access-kwrz5 for pod openshift-network-diagnostics/network-check-target-5xvng: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:55:16.805892 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:16.805787 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c9845ea-a9d6-4d5c-93ea-fb096666a2b9-kube-api-access-kwrz5 podName:3c9845ea-a9d6-4d5c-93ea-fb096666a2b9 nodeName:}" failed. No retries permitted until 2026-10-31 00:55:17.305770482 +0000 UTC m=+3.120969340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kwrz5" (UniqueName: "kubernetes.io/projected/3c9845ea-a9d6-4d5c-93ea-fb096666a2b9-kube-api-access-kwrz5") pod "network-check-target-5xvng" (UID: "3c9845ea-a9d6-4d5c-93ea-fb096666a2b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:55:16.807051 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.807027 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9v2m\" (UniqueName: \"kubernetes.io/projected/4b8ba0db-d8b9-4a40-a144-1400634cd99d-kube-api-access-t9v2m\") pod \"iptables-alerter-mtwcc\" (UID: \"4b8ba0db-d8b9-4a40-a144-1400634cd99d\") " pod="openshift-network-operator/iptables-alerter-mtwcc" Oct 31 00:55:16.807668 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.807646 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4qw7\" (UniqueName: \"kubernetes.io/projected/cd654994-98a0-4d39-aa38-270b4a0d4341-kube-api-access-m4qw7\") pod \"node-resolver-qsppl\" (UID: \"cd654994-98a0-4d39-aa38-270b4a0d4341\") " pod="openshift-dns/node-resolver-qsppl" Oct 31 00:55:16.808220 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.808195 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nchj\" (UniqueName: \"kubernetes.io/projected/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f-kube-api-access-7nchj\") pod \"network-metrics-daemon-pcvbh\" (UID: \"0fe8107c-9128-4c8a-9bde-e1cacd2ed19f\") " pod="openshift-multus/network-metrics-daemon-pcvbh" Oct 31 00:55:16.899888 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.899856 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qzm2t" Oct 31 00:55:16.903081 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.903049 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-p8vd6" Oct 31 00:55:16.913105 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:16.913077 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a37f05b_0755_4ca7_8d9b_12030234bec1.slice/crio-6bf194b52ad24d2798d0b7cd0a2cb4b697c1e7a6a27d53f708f38fadedcdfe34 WatchSource:0}: Error finding container 6bf194b52ad24d2798d0b7cd0a2cb4b697c1e7a6a27d53f708f38fadedcdfe34: Status 404 returned error can't find the container with id 6bf194b52ad24d2798d0b7cd0a2cb4b697c1e7a6a27d53f708f38fadedcdfe34 Oct 31 00:55:16.914417 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:16.914390 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88c1a03a_cb94_4226_9d53_cd923895569e.slice/crio-8bff4825a84b63e6edcadd67b0db93a359a938cb9e14f8c638eee7815a23cd8c WatchSource:0}: Error finding container 8bff4825a84b63e6edcadd67b0db93a359a938cb9e14f8c638eee7815a23cd8c: Status 404 returned error can't find the container with id 8bff4825a84b63e6edcadd67b0db93a359a938cb9e14f8c638eee7815a23cd8c Oct 31 00:55:16.925596 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.925576 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:55:16.933797 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.933773 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:16.933907 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.933842 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qsppl" Oct 31 00:55:16.936011 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.935993 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:16.938429 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.938409 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mtwcc" Oct 31 00:55:16.943607 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:16.943577 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3939885_f5d4_4bd2_a69f_435c2a8fa048.slice/crio-e819bed747b2de3500ac2a00a9d954a62304db15d628d5fbca0e83e9eafe8ad5 WatchSource:0}: Error finding container e819bed747b2de3500ac2a00a9d954a62304db15d628d5fbca0e83e9eafe8ad5: Status 404 returned error can't find the container with id e819bed747b2de3500ac2a00a9d954a62304db15d628d5fbca0e83e9eafe8ad5 Oct 31 00:55:16.947141 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:16.947111 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd654994_98a0_4d39_aa38_270b4a0d4341.slice/crio-4581a2b0496fd321747947f34a5456bc2a49e484303cb17f074217897d5eb5f2 WatchSource:0}: Error finding container 4581a2b0496fd321747947f34a5456bc2a49e484303cb17f074217897d5eb5f2: Status 404 returned error can't find the container with id 4581a2b0496fd321747947f34a5456bc2a49e484303cb17f074217897d5eb5f2 Oct 31 00:55:16.948528 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.948506 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8kln5" Oct 31 00:55:16.955773 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:16.955744 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b8ba0db_d8b9_4a40_a144_1400634cd99d.slice/crio-ebd6b5435799a1093f7e6e89ffbc0ba1b578811f81bd0a71c16f532a75fd826f WatchSource:0}: Error finding container ebd6b5435799a1093f7e6e89ffbc0ba1b578811f81bd0a71c16f532a75fd826f: Status 404 returned error can't find the container with id ebd6b5435799a1093f7e6e89ffbc0ba1b578811f81bd0a71c16f532a75fd826f Oct 31 00:55:16.959273 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.959243 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g5h4s" Oct 31 00:55:16.960759 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:16.960736 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12002bf6_fce5_4d20_a62c_6782522acd47.slice/crio-5f23efb80a3b70539d7eddbd4ef57b59f0b90f62c7f55e0509ac94d636ae5698 WatchSource:0}: Error finding container 5f23efb80a3b70539d7eddbd4ef57b59f0b90f62c7f55e0509ac94d636ae5698: Status 404 returned error can't find the container with id 5f23efb80a3b70539d7eddbd4ef57b59f0b90f62c7f55e0509ac94d636ae5698 Oct 31 00:55:16.965659 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:16.965636 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:16.971805 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:16.970548 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd610021a_d4d9_473d_8889_cb5afcb5b8a0.slice/crio-f3361e09423c13c5d72aa9f81c5fdd72aa2b6dea2802635f190aa391de176b5c WatchSource:0}: Error finding container f3361e09423c13c5d72aa9f81c5fdd72aa2b6dea2802635f190aa391de176b5c: Status 404 returned error can't find the container with id f3361e09423c13c5d72aa9f81c5fdd72aa2b6dea2802635f190aa391de176b5c Oct 31 00:55:16.975981 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:16.975954 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6ddfc5f_887b_4821_bac9_54cb59c50e32.slice/crio-4fc25dd6c3c416e147a16b29cc4c1b36fedb7a3b985955c578261141cf20a426 WatchSource:0}: Error finding container 4fc25dd6c3c416e147a16b29cc4c1b36fedb7a3b985955c578261141cf20a426: Status 404 returned error can't find the container with id 4fc25dd6c3c416e147a16b29cc4c1b36fedb7a3b985955c578261141cf20a426 Oct 31 00:55:17.201324 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:17.201221 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/49f8a438-6b29-4160-98e1-3f26e27fe0af-original-pull-secret\") pod \"global-pull-secret-syncer-s9tlh\" (UID: \"49f8a438-6b29-4160-98e1-3f26e27fe0af\") " pod="kube-system/global-pull-secret-syncer-s9tlh" Oct 31 00:55:17.201502 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:17.201376 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:55:17.201502 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:17.201453 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49f8a438-6b29-4160-98e1-3f26e27fe0af-original-pull-secret podName:49f8a438-6b29-4160-98e1-3f26e27fe0af nodeName:}" failed. No retries permitted until 2026-10-31 00:55:18.201433105 +0000 UTC m=+4.016631980 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/49f8a438-6b29-4160-98e1-3f26e27fe0af-original-pull-secret") pod "global-pull-secret-syncer-s9tlh" (UID: "49f8a438-6b29-4160-98e1-3f26e27fe0af") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:55:17.302487 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:17.302429 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f-metrics-certs\") pod \"network-metrics-daemon-pcvbh\" (UID: \"0fe8107c-9128-4c8a-9bde-e1cacd2ed19f\") " pod="openshift-multus/network-metrics-daemon-pcvbh" Oct 31 00:55:17.302672 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:17.302631 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:55:17.302749 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:17.302718 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f-metrics-certs podName:0fe8107c-9128-4c8a-9bde-e1cacd2ed19f nodeName:}" failed. No retries permitted until 2026-10-31 00:55:18.302697195 +0000 UTC m=+4.117896063 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f-metrics-certs") pod "network-metrics-daemon-pcvbh" (UID: "0fe8107c-9128-4c8a-9bde-e1cacd2ed19f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:55:17.403231 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:17.403192 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwrz5\" (UniqueName: \"kubernetes.io/projected/3c9845ea-a9d6-4d5c-93ea-fb096666a2b9-kube-api-access-kwrz5\") pod \"network-check-target-5xvng\" (UID: \"3c9845ea-a9d6-4d5c-93ea-fb096666a2b9\") " pod="openshift-network-diagnostics/network-check-target-5xvng" Oct 31 00:55:17.403411 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:17.403389 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:55:17.403411 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:17.403405 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:55:17.403551 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:17.403415 2576 projected.go:196] Error preparing data for projected volume kube-api-access-kwrz5 for pod openshift-network-diagnostics/network-check-target-5xvng: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:55:17.403551 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:17.403491 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c9845ea-a9d6-4d5c-93ea-fb096666a2b9-kube-api-access-kwrz5 podName:3c9845ea-a9d6-4d5c-93ea-fb096666a2b9 nodeName:}" failed. No retries permitted until 2026-10-31 00:55:18.403471648 +0000 UTC m=+4.218670516 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kwrz5" (UniqueName: "kubernetes.io/projected/3c9845ea-a9d6-4d5c-93ea-fb096666a2b9-kube-api-access-kwrz5") pod "network-check-target-5xvng" (UID: "3c9845ea-a9d6-4d5c-93ea-fb096666a2b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:55:17.650166 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:17.649601 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s9tlh" Oct 31 00:55:17.650166 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:17.649742 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s9tlh" podUID="49f8a438-6b29-4160-98e1-3f26e27fe0af" Oct 31 00:55:17.665646 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:17.665515 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" event={"ID":"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d","Type":"ContainerStarted","Data":"f26c472e51c7f868476b174a94eb7f7f44e59a358941d0a9d6133a1f25c73842"} Oct 31 00:55:17.676775 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:17.676738 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" event={"ID":"c3939885-f5d4-4bd2-a69f-435c2a8fa048","Type":"ContainerStarted","Data":"e819bed747b2de3500ac2a00a9d954a62304db15d628d5fbca0e83e9eafe8ad5"} Oct 31 00:55:17.681673 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:17.681594 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p8vd6" event={"ID":"88c1a03a-cb94-4226-9d53-cd923895569e","Type":"ContainerStarted","Data":"8bff4825a84b63e6edcadd67b0db93a359a938cb9e14f8c638eee7815a23cd8c"} Oct 31 00:55:17.719981 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:17.719944 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" event={"ID":"a6ddfc5f-887b-4821-bac9-54cb59c50e32","Type":"ContainerStarted","Data":"4fc25dd6c3c416e147a16b29cc4c1b36fedb7a3b985955c578261141cf20a426"} Oct 31 00:55:17.723440 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:17.723376 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:50:17 +0000 UTC" deadline="2028-04-29 01:51:37.885217996 +0000 UTC" Oct 31 00:55:17.723581 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:17.723448 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13104h56m20.161788465s" Oct 31 00:55:17.724139 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:17.724109 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g5h4s" event={"ID":"d610021a-d4d9-473d-8889-cb5afcb5b8a0","Type":"ContainerStarted","Data":"f3361e09423c13c5d72aa9f81c5fdd72aa2b6dea2802635f190aa391de176b5c"} Oct 31 00:55:17.737132 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:17.737103 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8kln5" event={"ID":"12002bf6-fce5-4d20-a62c-6782522acd47","Type":"ContainerStarted","Data":"5f23efb80a3b70539d7eddbd4ef57b59f0b90f62c7f55e0509ac94d636ae5698"} Oct 31 00:55:17.739108 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:17.739075 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mtwcc" event={"ID":"4b8ba0db-d8b9-4a40-a144-1400634cd99d","Type":"ContainerStarted","Data":"ebd6b5435799a1093f7e6e89ffbc0ba1b578811f81bd0a71c16f532a75fd826f"} Oct 31 00:55:17.744126 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:17.744071 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qsppl" event={"ID":"cd654994-98a0-4d39-aa38-270b4a0d4341","Type":"ContainerStarted","Data":"4581a2b0496fd321747947f34a5456bc2a49e484303cb17f074217897d5eb5f2"} Oct 31 00:55:17.753187 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:17.753156 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qzm2t" event={"ID":"0a37f05b-0755-4ca7-8d9b-12030234bec1","Type":"ContainerStarted","Data":"6bf194b52ad24d2798d0b7cd0a2cb4b697c1e7a6a27d53f708f38fadedcdfe34"} Oct 31 00:55:18.209448 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:18.209340 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/49f8a438-6b29-4160-98e1-3f26e27fe0af-original-pull-secret\") pod \"global-pull-secret-syncer-s9tlh\" (UID: \"49f8a438-6b29-4160-98e1-3f26e27fe0af\") " pod="kube-system/global-pull-secret-syncer-s9tlh" Oct 31 00:55:18.209629 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:18.209577 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:55:18.209703 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:18.209645 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49f8a438-6b29-4160-98e1-3f26e27fe0af-original-pull-secret podName:49f8a438-6b29-4160-98e1-3f26e27fe0af nodeName:}" failed. No retries permitted until 2026-10-31 00:55:20.209625559 +0000 UTC m=+6.024824434 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/49f8a438-6b29-4160-98e1-3f26e27fe0af-original-pull-secret") pod "global-pull-secret-syncer-s9tlh" (UID: "49f8a438-6b29-4160-98e1-3f26e27fe0af") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:55:18.310640 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:18.310589 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f-metrics-certs\") pod \"network-metrics-daemon-pcvbh\" (UID: \"0fe8107c-9128-4c8a-9bde-e1cacd2ed19f\") " pod="openshift-multus/network-metrics-daemon-pcvbh" Oct 31 00:55:18.318878 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:18.318850 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:55:18.319026 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:18.318939 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f-metrics-certs podName:0fe8107c-9128-4c8a-9bde-e1cacd2ed19f nodeName:}" failed. No retries permitted until 2026-10-31 00:55:20.318918396 +0000 UTC m=+6.134117247 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f-metrics-certs") pod "network-metrics-daemon-pcvbh" (UID: "0fe8107c-9128-4c8a-9bde-e1cacd2ed19f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:55:18.411980 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:18.411918 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwrz5\" (UniqueName: \"kubernetes.io/projected/3c9845ea-a9d6-4d5c-93ea-fb096666a2b9-kube-api-access-kwrz5\") pod \"network-check-target-5xvng\" (UID: \"3c9845ea-a9d6-4d5c-93ea-fb096666a2b9\") " pod="openshift-network-diagnostics/network-check-target-5xvng" Oct 31 00:55:18.412179 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:18.412106 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:55:18.412179 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:18.412133 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:55:18.412179 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:18.412147 2576 projected.go:196] Error preparing data for projected volume kube-api-access-kwrz5 for pod openshift-network-diagnostics/network-check-target-5xvng: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:55:18.412333 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:18.412207 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c9845ea-a9d6-4d5c-93ea-fb096666a2b9-kube-api-access-kwrz5 podName:3c9845ea-a9d6-4d5c-93ea-fb096666a2b9 nodeName:}" failed. No retries permitted until 2026-10-31 00:55:20.412187934 +0000 UTC m=+6.227386786 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kwrz5" (UniqueName: "kubernetes.io/projected/3c9845ea-a9d6-4d5c-93ea-fb096666a2b9-kube-api-access-kwrz5") pod "network-check-target-5xvng" (UID: "3c9845ea-a9d6-4d5c-93ea-fb096666a2b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:55:18.652423 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:18.652339 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xvng" Oct 31 00:55:18.653065 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:18.652487 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xvng" podUID="3c9845ea-a9d6-4d5c-93ea-fb096666a2b9" Oct 31 00:55:18.653065 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:18.652918 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pcvbh" Oct 31 00:55:18.653065 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:18.653034 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pcvbh" podUID="0fe8107c-9128-4c8a-9bde-e1cacd2ed19f" Oct 31 00:55:18.723838 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:18.723704 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:50:17 +0000 UTC" deadline="2028-07-24 02:37:40.118991672 +0000 UTC" Oct 31 00:55:18.723838 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:18.723766 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15169h42m21.395228758s" Oct 31 00:55:18.766228 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:18.766044 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-39.us-east-2.compute.internal" event={"ID":"eaafb8f2be56642836cc3cf93864aa97","Type":"ContainerStarted","Data":"e6a5b6c9a5074780e347cae94958840fdae61b0458a8fb76adbcbc4c166efbff"} Oct 31 00:55:19.648858 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:19.648820 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s9tlh" Oct 31 00:55:19.649118 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:19.648961 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s9tlh" podUID="49f8a438-6b29-4160-98e1-3f26e27fe0af" Oct 31 00:55:20.227917 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:20.227833 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/49f8a438-6b29-4160-98e1-3f26e27fe0af-original-pull-secret\") pod \"global-pull-secret-syncer-s9tlh\" (UID: \"49f8a438-6b29-4160-98e1-3f26e27fe0af\") " pod="kube-system/global-pull-secret-syncer-s9tlh" Oct 31 00:55:20.228363 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:20.228072 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:55:20.228363 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:20.228179 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49f8a438-6b29-4160-98e1-3f26e27fe0af-original-pull-secret podName:49f8a438-6b29-4160-98e1-3f26e27fe0af nodeName:}" failed. No retries permitted until 2026-10-31 00:55:24.228143496 +0000 UTC m=+10.043342351 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/49f8a438-6b29-4160-98e1-3f26e27fe0af-original-pull-secret") pod "global-pull-secret-syncer-s9tlh" (UID: "49f8a438-6b29-4160-98e1-3f26e27fe0af") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:55:20.328822 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:20.328784 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f-metrics-certs\") pod \"network-metrics-daemon-pcvbh\" (UID: \"0fe8107c-9128-4c8a-9bde-e1cacd2ed19f\") " pod="openshift-multus/network-metrics-daemon-pcvbh" Oct 31 00:55:20.329012 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:20.328975 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:55:20.329112 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:20.329047 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f-metrics-certs podName:0fe8107c-9128-4c8a-9bde-e1cacd2ed19f nodeName:}" failed. No retries permitted until 2026-10-31 00:55:24.32902572 +0000 UTC m=+10.144224596 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f-metrics-certs") pod "network-metrics-daemon-pcvbh" (UID: "0fe8107c-9128-4c8a-9bde-e1cacd2ed19f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:55:20.429633 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:20.429222 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwrz5\" (UniqueName: \"kubernetes.io/projected/3c9845ea-a9d6-4d5c-93ea-fb096666a2b9-kube-api-access-kwrz5\") pod \"network-check-target-5xvng\" (UID: \"3c9845ea-a9d6-4d5c-93ea-fb096666a2b9\") " pod="openshift-network-diagnostics/network-check-target-5xvng" Oct 31 00:55:20.429633 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:20.429364 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:55:20.429633 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:20.429386 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:55:20.429633 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:20.429398 2576 projected.go:196] Error preparing data for projected volume kube-api-access-kwrz5 for pod openshift-network-diagnostics/network-check-target-5xvng: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:55:20.429633 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:20.429448 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c9845ea-a9d6-4d5c-93ea-fb096666a2b9-kube-api-access-kwrz5 podName:3c9845ea-a9d6-4d5c-93ea-fb096666a2b9 nodeName:}" failed. No retries permitted until 2026-10-31 00:55:24.42943381 +0000 UTC m=+10.244632677 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kwrz5" (UniqueName: "kubernetes.io/projected/3c9845ea-a9d6-4d5c-93ea-fb096666a2b9-kube-api-access-kwrz5") pod "network-check-target-5xvng" (UID: "3c9845ea-a9d6-4d5c-93ea-fb096666a2b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:55:20.651750 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:20.651516 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xvng" Oct 31 00:55:20.651888 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:20.651862 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xvng" podUID="3c9845ea-a9d6-4d5c-93ea-fb096666a2b9" Oct 31 00:55:20.651974 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:20.651942 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pcvbh" Oct 31 00:55:20.652065 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:20.652035 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pcvbh" podUID="0fe8107c-9128-4c8a-9bde-e1cacd2ed19f" Oct 31 00:55:20.779169 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:20.779085 2576 generic.go:356] "Generic (PLEG): container finished" podID="88c1a03a-cb94-4226-9d53-cd923895569e" containerID="e0aa8d81ae98b35c2c8df1c330db064a0fd8180b04556aefbdcfcc556bb9d5c4" exitCode=0 Oct 31 00:55:20.779419 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:20.779396 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p8vd6" event={"ID":"88c1a03a-cb94-4226-9d53-cd923895569e","Type":"ContainerDied","Data":"e0aa8d81ae98b35c2c8df1c330db064a0fd8180b04556aefbdcfcc556bb9d5c4"} Oct 31 00:55:20.783475 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:20.783454 2576 generic.go:356] "Generic (PLEG): container finished" podID="ce6a902de7718819f66be8130076b8e6" containerID="4e433a1feb85eed73620569224ecdabfb4a5ce29eb2412c13506fd108fd8dbf2" exitCode=0 Oct 31 00:55:20.783589 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:20.783485 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal" event={"ID":"ce6a902de7718819f66be8130076b8e6","Type":"ContainerDied","Data":"4e433a1feb85eed73620569224ecdabfb4a5ce29eb2412c13506fd108fd8dbf2"} Oct 31 00:55:20.807240 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:20.805968 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-39.us-east-2.compute.internal" podStartSLOduration=4.805950847 podStartE2EDuration="4.805950847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:18.781896481 +0000 UTC m=+4.597095358" watchObservedRunningTime="2026-10-31 00:55:20.805950847 +0000 UTC m=+6.621149724" Oct 31 00:55:21.649166 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:21.649130 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s9tlh" Oct 31 00:55:21.649924 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:21.649278 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s9tlh" podUID="49f8a438-6b29-4160-98e1-3f26e27fe0af" Oct 31 00:55:21.791103 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:21.790303 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal" event={"ID":"ce6a902de7718819f66be8130076b8e6","Type":"ContainerStarted","Data":"e3611575ec54eb27b687d7eafd5116a93e852f66843821402196e9b29faab55f"} Oct 31 00:55:22.652018 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:22.651982 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pcvbh" Oct 31 00:55:22.652585 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:22.652134 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pcvbh" podUID="0fe8107c-9128-4c8a-9bde-e1cacd2ed19f" Oct 31 00:55:22.652673 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:22.652586 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xvng" Oct 31 00:55:22.652734 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:22.652694 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xvng" podUID="3c9845ea-a9d6-4d5c-93ea-fb096666a2b9" Oct 31 00:55:22.799258 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:22.799201 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8kln5" event={"ID":"12002bf6-fce5-4d20-a62c-6782522acd47","Type":"ContainerStarted","Data":"03bfad0912861dfa88dc5b8aad458838e72777db49e527b4b8de6dc1617dfaba"} Oct 31 00:55:22.815629 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:22.815560 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal" podStartSLOduration=6.815539744 podStartE2EDuration="6.815539744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:21.816765163 +0000 UTC m=+7.631964041" watchObservedRunningTime="2026-10-31 00:55:22.815539744 +0000 UTC m=+8.630738621" Oct 31 00:55:22.815861 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:22.815736 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8kln5" podStartSLOduration=2.781602548 podStartE2EDuration="7.815725112s" totalImagesPullingTime="5.034122564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:55:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:22.814660391 +0000 UTC m=+8.629859272" watchObservedRunningTime="2026-10-31 00:55:22.815725112 +0000 UTC m=+8.630923990" Oct 31 00:55:23.650301 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:23.649769 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s9tlh" Oct 31 00:55:23.650301 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:23.649904 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s9tlh" podUID="49f8a438-6b29-4160-98e1-3f26e27fe0af" Oct 31 00:55:23.802721 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:23.801973 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qzm2t" event={"ID":"0a37f05b-0755-4ca7-8d9b-12030234bec1","Type":"ContainerStarted","Data":"e0132390cb829219edd1798d315c209ff58e811fac4e499326e717fe94cdd7e8"} Oct 31 00:55:23.806247 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:23.805464 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" event={"ID":"c3939885-f5d4-4bd2-a69f-435c2a8fa048","Type":"ContainerStarted","Data":"f19e0ce87a22ec4dee9138fa7e3659f7d26ed72a6f27a572c19230bc0fc8b2f1"} Oct 31 00:55:24.261725 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:24.261686 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/49f8a438-6b29-4160-98e1-3f26e27fe0af-original-pull-secret\") pod \"global-pull-secret-syncer-s9tlh\" (UID: \"49f8a438-6b29-4160-98e1-3f26e27fe0af\") " pod="kube-system/global-pull-secret-syncer-s9tlh" Oct 31 00:55:24.261917 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:24.261895 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:55:24.261986 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:24.261974 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49f8a438-6b29-4160-98e1-3f26e27fe0af-original-pull-secret podName:49f8a438-6b29-4160-98e1-3f26e27fe0af nodeName:}" failed. No retries permitted until 2026-10-31 00:55:32.261951565 +0000 UTC m=+18.077150419 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/49f8a438-6b29-4160-98e1-3f26e27fe0af-original-pull-secret") pod "global-pull-secret-syncer-s9tlh" (UID: "49f8a438-6b29-4160-98e1-3f26e27fe0af") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:55:24.362177 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:24.362091 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f-metrics-certs\") pod \"network-metrics-daemon-pcvbh\" (UID: \"0fe8107c-9128-4c8a-9bde-e1cacd2ed19f\") " pod="openshift-multus/network-metrics-daemon-pcvbh" Oct 31 00:55:24.362373 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:24.362299 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:55:24.362373 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:24.362364 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f-metrics-certs podName:0fe8107c-9128-4c8a-9bde-e1cacd2ed19f nodeName:}" failed. No retries permitted until 2026-10-31 00:55:32.362345646 +0000 UTC m=+18.177544501 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f-metrics-certs") pod "network-metrics-daemon-pcvbh" (UID: "0fe8107c-9128-4c8a-9bde-e1cacd2ed19f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:55:24.462970 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:24.462613 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwrz5\" (UniqueName: \"kubernetes.io/projected/3c9845ea-a9d6-4d5c-93ea-fb096666a2b9-kube-api-access-kwrz5\") pod \"network-check-target-5xvng\" (UID: \"3c9845ea-a9d6-4d5c-93ea-fb096666a2b9\") " pod="openshift-network-diagnostics/network-check-target-5xvng" Oct 31 00:55:24.462970 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:24.462728 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:55:24.462970 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:24.462744 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:55:24.462970 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:24.462752 2576 projected.go:196] Error preparing data for projected volume kube-api-access-kwrz5 for pod openshift-network-diagnostics/network-check-target-5xvng: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:55:24.462970 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:24.462801 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c9845ea-a9d6-4d5c-93ea-fb096666a2b9-kube-api-access-kwrz5 podName:3c9845ea-a9d6-4d5c-93ea-fb096666a2b9 nodeName:}" failed. No retries permitted until 2026-10-31 00:55:32.462789299 +0000 UTC m=+18.277988150 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kwrz5" (UniqueName: "kubernetes.io/projected/3c9845ea-a9d6-4d5c-93ea-fb096666a2b9-kube-api-access-kwrz5") pod "network-check-target-5xvng" (UID: "3c9845ea-a9d6-4d5c-93ea-fb096666a2b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:55:24.650535 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:24.649908 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xvng" Oct 31 00:55:24.650535 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:24.650049 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xvng" podUID="3c9845ea-a9d6-4d5c-93ea-fb096666a2b9" Oct 31 00:55:24.651032 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:24.650895 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pcvbh" Oct 31 00:55:24.651032 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:24.650984 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pcvbh" podUID="0fe8107c-9128-4c8a-9bde-e1cacd2ed19f" Oct 31 00:55:24.813482 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:24.813453 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8kln5" Oct 31 00:55:24.814657 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:24.814464 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8kln5" Oct 31 00:55:24.828683 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:24.828635 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qzm2t" podStartSLOduration=3.771581549 podStartE2EDuration="9.828621809s" totalImagesPullingTime="6.05704026s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:55:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:23.817190654 +0000 UTC m=+9.632389548" watchObservedRunningTime="2026-10-31 00:55:24.828621809 +0000 UTC m=+10.643820687" Oct 31 00:55:25.549059 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:25.548750 2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 00:55:25.649198 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:25.649168 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s9tlh" Oct 31 00:55:25.649360 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:25.649311 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s9tlh" podUID="49f8a438-6b29-4160-98e1-3f26e27fe0af" Oct 31 00:55:25.658310 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:25.658198 2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T00:55:25.548773928Z","UUID":"c71b2e15-0f17-4e67-9dc6-4eed8c521355","Handler":null,"Name":"","Endpoint":""} Oct 31 00:55:25.660008 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:25.659977 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 00:55:25.660008 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:25.660010 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 00:55:25.811316 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:25.811279 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" event={"ID":"c3939885-f5d4-4bd2-a69f-435c2a8fa048","Type":"ContainerStarted","Data":"4946011c28675e25c0b67e2ce6ecd5618049e1227672151aabbee44a81f48d78"} Oct 31 00:55:25.813360 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:25.813329 2576 generic.go:356] "Generic (PLEG): container finished" podID="88c1a03a-cb94-4226-9d53-cd923895569e" containerID="803a696233b5fef7d4370215e02d38f4aed242cbcf59290b6e750c9f4fac8097" exitCode=0 Oct 31 00:55:25.813517 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:25.813413 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p8vd6" event={"ID":"88c1a03a-cb94-4226-9d53-cd923895569e","Type":"ContainerDied","Data":"803a696233b5fef7d4370215e02d38f4aed242cbcf59290b6e750c9f4fac8097"} Oct 31 00:55:25.815382 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:25.815357 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g5h4s" event={"ID":"d610021a-d4d9-473d-8889-cb5afcb5b8a0","Type":"ContainerStarted","Data":"e9537a63bb63dc5fd5abee1fb337deb1a44ecb50a15682e63f3a7538f3fb5850"} Oct 31 00:55:25.817035 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:25.817013 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qsppl" event={"ID":"cd654994-98a0-4d39-aa38-270b4a0d4341","Type":"ContainerStarted","Data":"a64966f04e467a133dbca0946f38b9a3da764778525f18ab992e1e8439e10fc6"} Oct 31 00:55:25.862666 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:25.862615 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-g5h4s" podStartSLOduration=2.839712268 podStartE2EDuration="10.862597999s" totalImagesPullingTime="8.022885731s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:55:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:25.862288636 +0000 UTC m=+11.677487509" watchObservedRunningTime="2026-10-31 00:55:25.862597999 +0000 UTC m=+11.677796874" Oct 31 00:55:25.863075 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:25.863032 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-qsppl" podStartSLOduration=2.462643598 podStartE2EDuration="10.863016176s" totalImagesPullingTime="8.400372578s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:55:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:25.846340622 +0000 UTC m=+11.661539505" watchObservedRunningTime="2026-10-31 00:55:25.863016176 +0000 UTC m=+11.678215050" Oct 31 00:55:26.649768 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:26.649728 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xvng" Oct 31 00:55:26.649938 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:26.649868 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xvng" podUID="3c9845ea-a9d6-4d5c-93ea-fb096666a2b9" Oct 31 00:55:26.650232 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:26.650205 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pcvbh" Oct 31 00:55:26.650359 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:26.650305 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pcvbh" podUID="0fe8107c-9128-4c8a-9bde-e1cacd2ed19f" Oct 31 00:55:27.649571 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:27.649277 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s9tlh" Oct 31 00:55:27.650038 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:27.649661 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s9tlh" podUID="49f8a438-6b29-4160-98e1-3f26e27fe0af" Oct 31 00:55:27.829531 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:27.828904 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" event={"ID":"a6ddfc5f-887b-4821-bac9-54cb59c50e32","Type":"ContainerStarted","Data":"8452bf378ecb353f57d7015bdc1bd9aa601d0133ec12ac76f98203d26dffa09e"} Oct 31 00:55:27.832133 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:27.831643 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mtwcc" event={"ID":"4b8ba0db-d8b9-4a40-a144-1400634cd99d","Type":"ContainerStarted","Data":"7c64a282b015ea2c4eba1255beec37d24ba58a94998c9f5228bb927d68abee66"} Oct 31 00:55:27.864903 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:27.864854 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" podStartSLOduration=2.134057232 podStartE2EDuration="12.864839981s" totalImagesPullingTime="10.730782749s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:55:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:27.851459917 +0000 UTC m=+13.666658802" watchObservedRunningTime="2026-10-31 00:55:27.864839981 +0000 UTC m=+13.680038904" Oct 31 00:55:27.865094 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:27.865069 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-mtwcc" podStartSLOduration=4.387617201 podStartE2EDuration="12.865060729s" totalImagesPullingTime="8.477443528s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:55:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:27.864695021 +0000 UTC m=+13.679893894" watchObservedRunningTime="2026-10-31 00:55:27.865060729 +0000 UTC m=+13.680259603" Oct 31 00:55:28.649245 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:28.649209 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pcvbh" Oct 31 00:55:28.649245 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:28.649243 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xvng" Oct 31 00:55:28.649465 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:28.649355 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pcvbh" podUID="0fe8107c-9128-4c8a-9bde-e1cacd2ed19f" Oct 31 00:55:28.649542 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:28.649498 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xvng" podUID="3c9845ea-a9d6-4d5c-93ea-fb096666a2b9" Oct 31 00:55:28.836123 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:28.836088 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" event={"ID":"c3939885-f5d4-4bd2-a69f-435c2a8fa048","Type":"ContainerStarted","Data":"762eb04382d9b4e07d9a2ebf7138b634603809a772c8bb3aa40158604210c7ed"} Oct 31 00:55:28.838233 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:28.838209 2576 generic.go:356] "Generic (PLEG): container finished" podID="88c1a03a-cb94-4226-9d53-cd923895569e" containerID="0090b900cfe0d52499a44a37f4ec26fd067884b6f9d958abc1bd0c4592521d4e" exitCode=0 Oct 31 00:55:28.838325 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:28.838289 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p8vd6" event={"ID":"88c1a03a-cb94-4226-9d53-cd923895569e","Type":"ContainerDied","Data":"0090b900cfe0d52499a44a37f4ec26fd067884b6f9d958abc1bd0c4592521d4e"} Oct 31 00:55:28.854047 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:28.854005 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" podStartSLOduration=3.510317828 podStartE2EDuration="13.853993424s" totalImagesPullingTime="10.343675596s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:55:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:28.852527088 +0000 UTC m=+14.667725960" watchObservedRunningTime="2026-10-31 00:55:28.853993424 +0000 UTC m=+14.669192293" Oct 31 00:55:29.649142 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:29.649100 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s9tlh" Oct 31 00:55:29.649300 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:29.649252 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s9tlh" podUID="49f8a438-6b29-4160-98e1-3f26e27fe0af" Oct 31 00:55:30.649244 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:30.649206 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pcvbh" Oct 31 00:55:30.649953 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:30.649258 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xvng" Oct 31 00:55:30.649953 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:30.649382 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pcvbh" podUID="0fe8107c-9128-4c8a-9bde-e1cacd2ed19f" Oct 31 00:55:30.649953 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:30.649510 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xvng" podUID="3c9845ea-a9d6-4d5c-93ea-fb096666a2b9" Oct 31 00:55:30.845684 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:30.845639 2576 generic.go:356] "Generic (PLEG): container finished" podID="88c1a03a-cb94-4226-9d53-cd923895569e" containerID="cf91db1cbe3eeb915ca5b3ac11916ff33583863a26368d54e828df2b03e07a9f" exitCode=0 Oct 31 00:55:30.845837 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:30.845708 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p8vd6" event={"ID":"88c1a03a-cb94-4226-9d53-cd923895569e","Type":"ContainerDied","Data":"cf91db1cbe3eeb915ca5b3ac11916ff33583863a26368d54e828df2b03e07a9f"} Oct 31 00:55:31.649735 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:31.649702 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s9tlh" Oct 31 00:55:31.650225 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:31.649836 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s9tlh" podUID="49f8a438-6b29-4160-98e1-3f26e27fe0af" Oct 31 00:55:32.324468 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:32.324221 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/49f8a438-6b29-4160-98e1-3f26e27fe0af-original-pull-secret\") pod \"global-pull-secret-syncer-s9tlh\" (UID: \"49f8a438-6b29-4160-98e1-3f26e27fe0af\") " pod="kube-system/global-pull-secret-syncer-s9tlh" Oct 31 00:55:32.324645 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:32.324376 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:55:32.324645 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:32.324612 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49f8a438-6b29-4160-98e1-3f26e27fe0af-original-pull-secret podName:49f8a438-6b29-4160-98e1-3f26e27fe0af nodeName:}" failed. No retries permitted until 2026-10-31 00:55:48.324594903 +0000 UTC m=+34.139793757 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/49f8a438-6b29-4160-98e1-3f26e27fe0af-original-pull-secret") pod "global-pull-secret-syncer-s9tlh" (UID: "49f8a438-6b29-4160-98e1-3f26e27fe0af") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:55:32.425185 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:32.425148 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f-metrics-certs\") pod \"network-metrics-daemon-pcvbh\" (UID: \"0fe8107c-9128-4c8a-9bde-e1cacd2ed19f\") " pod="openshift-multus/network-metrics-daemon-pcvbh" Oct 31 00:55:32.425363 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:32.425315 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:55:32.425438 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:32.425371 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f-metrics-certs podName:0fe8107c-9128-4c8a-9bde-e1cacd2ed19f nodeName:}" failed. No retries permitted until 2026-10-31 00:55:48.425354902 +0000 UTC m=+34.240553767 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f-metrics-certs") pod "network-metrics-daemon-pcvbh" (UID: "0fe8107c-9128-4c8a-9bde-e1cacd2ed19f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:55:32.526302 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:32.526262 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwrz5\" (UniqueName: \"kubernetes.io/projected/3c9845ea-a9d6-4d5c-93ea-fb096666a2b9-kube-api-access-kwrz5\") pod \"network-check-target-5xvng\" (UID: \"3c9845ea-a9d6-4d5c-93ea-fb096666a2b9\") " pod="openshift-network-diagnostics/network-check-target-5xvng" Oct 31 00:55:32.526467 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:32.526443 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:55:32.526467 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:32.526468 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:55:32.526583 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:32.526482 2576 projected.go:196] Error preparing data for projected volume kube-api-access-kwrz5 for pod openshift-network-diagnostics/network-check-target-5xvng: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:55:32.526583 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:32.526548 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c9845ea-a9d6-4d5c-93ea-fb096666a2b9-kube-api-access-kwrz5 podName:3c9845ea-a9d6-4d5c-93ea-fb096666a2b9 nodeName:}" failed. No retries permitted until 2026-10-31 00:55:48.526529397 +0000 UTC m=+34.341728259 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kwrz5" (UniqueName: "kubernetes.io/projected/3c9845ea-a9d6-4d5c-93ea-fb096666a2b9-kube-api-access-kwrz5") pod "network-check-target-5xvng" (UID: "3c9845ea-a9d6-4d5c-93ea-fb096666a2b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:55:32.649962 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:32.649852 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xvng" Oct 31 00:55:32.650389 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:32.649978 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xvng" podUID="3c9845ea-a9d6-4d5c-93ea-fb096666a2b9" Oct 31 00:55:32.650472 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:32.650391 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pcvbh" Oct 31 00:55:32.650541 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:32.650514 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pcvbh" podUID="0fe8107c-9128-4c8a-9bde-e1cacd2ed19f" Oct 31 00:55:32.852347 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:32.852316 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p8vd6" event={"ID":"88c1a03a-cb94-4226-9d53-cd923895569e","Type":"ContainerStarted","Data":"db051957365c06b73c8dda860f6f158e54f9862e6de96cc1a3b09cf93518ccc2"} Oct 31 00:55:33.649086 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:33.649051 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s9tlh" Oct 31 00:55:33.649230 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:33.649161 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s9tlh" podUID="49f8a438-6b29-4160-98e1-3f26e27fe0af" Oct 31 00:55:33.857628 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:33.857413 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" event={"ID":"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d","Type":"ContainerStarted","Data":"d427c7e131c65c1c908713b257314a5204b927b3f57ecbc6c6262e9fb7849a7b"} Oct 31 00:55:33.858130 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:33.857684 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" event={"ID":"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d","Type":"ContainerStarted","Data":"40b71395e2400b4f992be9b3e0f439589cc04c82928c2fe8b99d058569009126"} Oct 31 00:55:33.858130 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:33.857725 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" event={"ID":"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d","Type":"ContainerStarted","Data":"5aa9a56b98f83713a95905d4af6ce3001e35bfcf6245833caf0abdd79f47a65d"} Oct 31 00:55:33.858130 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:33.857756 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" event={"ID":"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d","Type":"ContainerStarted","Data":"c5fe7bad44e59e6f1c88eab57dc53bfe2203f5af5710ec348d9efdd4b82c2e12"} Oct 31 00:55:33.858130 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:33.857797 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" event={"ID":"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d","Type":"ContainerStarted","Data":"9ae7a19510067127f27b6e7498912b246fff3dbbbec6ada840493876178631b6"} Oct 31 00:55:33.858130 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:33.857825 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" event={"ID":"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d","Type":"ContainerStarted","Data":"48580cf2a38303053c9dd585734ff7a03fccb51a980599c6400c3621e86d6074"} Oct 31 00:55:33.860074 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:33.860041 2576 generic.go:356] "Generic (PLEG): container finished" podID="88c1a03a-cb94-4226-9d53-cd923895569e" containerID="db051957365c06b73c8dda860f6f158e54f9862e6de96cc1a3b09cf93518ccc2" exitCode=0 Oct 31 00:55:33.860195 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:33.860082 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p8vd6" event={"ID":"88c1a03a-cb94-4226-9d53-cd923895569e","Type":"ContainerDied","Data":"db051957365c06b73c8dda860f6f158e54f9862e6de96cc1a3b09cf93518ccc2"} Oct 31 00:55:34.649688 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:34.649647 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xvng" Oct 31 00:55:34.649868 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:34.649836 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xvng" podUID="3c9845ea-a9d6-4d5c-93ea-fb096666a2b9" Oct 31 00:55:34.649927 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:34.649880 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pcvbh" Oct 31 00:55:34.650009 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:34.649992 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pcvbh" podUID="0fe8107c-9128-4c8a-9bde-e1cacd2ed19f" Oct 31 00:55:34.865429 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:34.865387 2576 generic.go:356] "Generic (PLEG): container finished" podID="88c1a03a-cb94-4226-9d53-cd923895569e" containerID="04099c4cd5dfe5c4572cd656ebf34bb77c5da6522bdae6a84444b7ee45bfb9c4" exitCode=0 Oct 31 00:55:34.865957 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:34.865448 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p8vd6" event={"ID":"88c1a03a-cb94-4226-9d53-cd923895569e","Type":"ContainerDied","Data":"04099c4cd5dfe5c4572cd656ebf34bb77c5da6522bdae6a84444b7ee45bfb9c4"} Oct 31 00:55:35.648840 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:35.648807 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s9tlh" Oct 31 00:55:35.648977 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:35.648912 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s9tlh" podUID="49f8a438-6b29-4160-98e1-3f26e27fe0af" Oct 31 00:55:35.871080 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:35.870984 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" event={"ID":"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d","Type":"ContainerStarted","Data":"6a0fd7b5d40c48a06ad4b489024573d2c32bb325e92254c2a275cc65b35a8341"} Oct 31 00:55:35.874186 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:35.874159 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p8vd6" event={"ID":"88c1a03a-cb94-4226-9d53-cd923895569e","Type":"ContainerStarted","Data":"14f08dbbcbe5541e81cb6feb92c124c9767832ea28bc1c7e6b37dc70a8562b58"} Oct 31 00:55:35.894104 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:35.894059 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-p8vd6" podStartSLOduration=8.62463965 podStartE2EDuration="20.89404443s" totalImagesPullingTime="12.26940478s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:55:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:35.892912897 +0000 UTC m=+21.708111782" watchObservedRunningTime="2026-10-31 00:55:35.89404443 +0000 UTC m=+21.709243303" Oct 31 00:55:36.649433 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:36.649392 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pcvbh" Oct 31 00:55:36.649628 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:36.649519 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pcvbh" podUID="0fe8107c-9128-4c8a-9bde-e1cacd2ed19f" Oct 31 00:55:36.649628 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:36.649567 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xvng" Oct 31 00:55:36.649737 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:36.649657 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xvng" podUID="3c9845ea-a9d6-4d5c-93ea-fb096666a2b9" Oct 31 00:55:37.649040 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:37.649017 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s9tlh" Oct 31 00:55:37.649466 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:37.649113 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s9tlh" podUID="49f8a438-6b29-4160-98e1-3f26e27fe0af" Oct 31 00:55:37.881682 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:37.881602 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" event={"ID":"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d","Type":"ContainerStarted","Data":"63bc28ef838cd8b7015450e6d2321b76cc0d39fe0b7bba327a3d502a7156c3da"} Oct 31 00:55:37.881898 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:37.881885 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:37.882148 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:37.882025 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:37.882148 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:37.882098 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:37.896178 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:37.896158 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:37.896276 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:37.896256 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:37.910944 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:37.910908 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" podStartSLOduration=6.706156975 podStartE2EDuration="22.910888459s" totalImagesPullingTime="16.204731484s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:55:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:37.910739272 +0000 UTC m=+23.725938157" watchObservedRunningTime="2026-10-31 00:55:37.910888459 +0000 UTC m=+23.726087358" Oct 31 00:55:38.649842 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:38.649805 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xvng" Oct 31 00:55:38.649842 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:38.649826 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pcvbh" Oct 31 00:55:38.650462 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:38.649914 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xvng" podUID="3c9845ea-a9d6-4d5c-93ea-fb096666a2b9" Oct 31 00:55:38.650462 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:38.649992 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pcvbh" podUID="0fe8107c-9128-4c8a-9bde-e1cacd2ed19f" Oct 31 00:55:39.387946 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:39.387633 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5xvng"] Oct 31 00:55:39.388070 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:39.387969 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xvng" Oct 31 00:55:39.388070 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:39.388056 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xvng" podUID="3c9845ea-a9d6-4d5c-93ea-fb096666a2b9" Oct 31 00:55:39.391010 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:39.390981 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-s9tlh"] Oct 31 00:55:39.391149 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:39.391086 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s9tlh" Oct 31 00:55:39.391195 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:39.391168 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s9tlh" podUID="49f8a438-6b29-4160-98e1-3f26e27fe0af" Oct 31 00:55:39.391831 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:39.391809 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pcvbh"] Oct 31 00:55:39.391915 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:39.391891 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pcvbh" Oct 31 00:55:39.391991 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:39.391973 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pcvbh" podUID="0fe8107c-9128-4c8a-9bde-e1cacd2ed19f" Oct 31 00:55:40.649730 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:40.649685 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s9tlh" Oct 31 00:55:40.650156 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:40.649828 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s9tlh" podUID="49f8a438-6b29-4160-98e1-3f26e27fe0af" Oct 31 00:55:40.650156 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:40.649880 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xvng" Oct 31 00:55:40.650156 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:40.649999 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xvng" podUID="3c9845ea-a9d6-4d5c-93ea-fb096666a2b9" Oct 31 00:55:41.649082 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:41.649044 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pcvbh" Oct 31 00:55:41.649341 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:41.649190 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pcvbh" podUID="0fe8107c-9128-4c8a-9bde-e1cacd2ed19f" Oct 31 00:55:42.649367 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:42.649290 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xvng" Oct 31 00:55:42.649367 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:42.649325 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s9tlh" Oct 31 00:55:42.649769 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:42.649409 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xvng" podUID="3c9845ea-a9d6-4d5c-93ea-fb096666a2b9" Oct 31 00:55:42.649769 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:42.649538 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s9tlh" podUID="49f8a438-6b29-4160-98e1-3f26e27fe0af" Oct 31 00:55:43.649616 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:43.649525 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pcvbh" Oct 31 00:55:43.650041 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:43.649653 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pcvbh" podUID="0fe8107c-9128-4c8a-9bde-e1cacd2ed19f" Oct 31 00:55:44.650048 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:44.649964 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s9tlh" Oct 31 00:55:44.650489 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:44.650051 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s9tlh" podUID="49f8a438-6b29-4160-98e1-3f26e27fe0af" Oct 31 00:55:44.650489 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:44.650123 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xvng" Oct 31 00:55:44.650602 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:44.650541 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xvng" podUID="3c9845ea-a9d6-4d5c-93ea-fb096666a2b9" Oct 31 00:55:45.155305 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:45.155274 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:45.314603 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:45.314576 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:45.343568 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:45.343535 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8kln5" Oct 31 00:55:45.343713 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:45.343697 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 00:55:45.344243 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:45.344219 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8kln5" Oct 31 00:55:45.452958 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:45.452932 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:45.648873 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:45.648840 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pcvbh" Oct 31 00:55:45.649029 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:45.648948 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pcvbh" podUID="0fe8107c-9128-4c8a-9bde-e1cacd2ed19f" Oct 31 00:55:46.406554 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:46.406523 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:46.467952 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:46.467927 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-39.us-east-2.compute.internal" event="NodeReady" Oct 31 00:55:46.468090 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:46.468058 2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 00:55:46.517393 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:46.517359 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zhmhj"] Oct 31 00:55:46.533974 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:46.533939 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:46.573320 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:46.573280 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-srw8h"] Oct 31 00:55:46.573517 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:46.573410 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zhmhj" Oct 31 00:55:46.576898 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:46.576875 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 00:55:46.577124 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:46.577107 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 00:55:46.577218 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:46.577195 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-72bs2\"" Oct 31 00:55:46.584126 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:46.583016 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zhmhj"] Oct 31 00:55:46.584126 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:46.583088 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-srw8h"] Oct 31 00:55:46.584126 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:46.583195 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-srw8h" Oct 31 00:55:46.589196 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:46.589176 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-58wnq\"" Oct 31 00:55:46.624269 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:46.624193 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49f7c8f4-b979-436d-b035-20f9fcae140a-config-volume\") pod \"dns-default-zhmhj\" (UID: \"49f7c8f4-b979-436d-b035-20f9fcae140a\") " pod="openshift-dns/dns-default-zhmhj" Oct 31 00:55:46.624269 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:46.624233 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzddr\" (UniqueName: \"kubernetes.io/projected/49f7c8f4-b979-436d-b035-20f9fcae140a-kube-api-access-vzddr\") pod \"dns-default-zhmhj\" (UID: \"49f7c8f4-b979-436d-b035-20f9fcae140a\") " pod="openshift-dns/dns-default-zhmhj" Oct 31 00:55:46.624269 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:46.624262 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49f7c8f4-b979-436d-b035-20f9fcae140a-tmp-dir\") pod \"dns-default-zhmhj\" (UID: \"49f7c8f4-b979-436d-b035-20f9fcae140a\") " pod="openshift-dns/dns-default-zhmhj" Oct 31 00:55:46.624549 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:46.624403 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49f7c8f4-b979-436d-b035-20f9fcae140a-metrics-tls\") pod \"dns-default-zhmhj\" (UID: \"49f7c8f4-b979-436d-b035-20f9fcae140a\") " pod="openshift-dns/dns-default-zhmhj" Oct 31 00:55:46.649061 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:46.649035 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s9tlh" Oct 31 00:55:46.649215 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:46.649194 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xvng" Oct 31 00:55:46.677251 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:46.677222 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-xb7hg\"" Oct 31 00:55:46.677251 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:46.677251 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 00:55:46.677463 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:46.677269 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-gczx4\"" Oct 31 00:55:46.677463 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:46.677373 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 00:55:46.679928 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:46.679903 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 00:55:46.680036 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:46.679955 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 00:55:46.680036 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:46.679973 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 00:55:46.680036 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:46.679994 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 00:55:46.725299 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:46.725262 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzddr\" (UniqueName: \"kubernetes.io/projected/49f7c8f4-b979-436d-b035-20f9fcae140a-kube-api-access-vzddr\") pod \"dns-default-zhmhj\" (UID: \"49f7c8f4-b979-436d-b035-20f9fcae140a\") " pod="openshift-dns/dns-default-zhmhj" Oct 31 00:55:46.725299 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:46.725306 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a11b541-6b88-439e-bcbe-5901cb4c20b0-cert\") pod \"ingress-canary-srw8h\" (UID: \"9a11b541-6b88-439e-bcbe-5901cb4c20b0\") " pod="openshift-ingress-canary/ingress-canary-srw8h" Oct 31 00:55:46.725539 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:46.725506 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49f7c8f4-b979-436d-b035-20f9fcae140a-tmp-dir\") pod \"dns-default-zhmhj\" (UID: \"49f7c8f4-b979-436d-b035-20f9fcae140a\") " pod="openshift-dns/dns-default-zhmhj" Oct 31 00:55:46.725600 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:46.725546 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnc9j\" (UniqueName: \"kubernetes.io/projected/9a11b541-6b88-439e-bcbe-5901cb4c20b0-kube-api-access-hnc9j\") pod \"ingress-canary-srw8h\" (UID: \"9a11b541-6b88-439e-bcbe-5901cb4c20b0\") " pod="openshift-ingress-canary/ingress-canary-srw8h" Oct 31 00:55:46.725600 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:46.725572 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49f7c8f4-b979-436d-b035-20f9fcae140a-metrics-tls\") pod \"dns-default-zhmhj\" (UID: \"49f7c8f4-b979-436d-b035-20f9fcae140a\") " pod="openshift-dns/dns-default-zhmhj" Oct 31 00:55:46.725689 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:46.725647 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49f7c8f4-b979-436d-b035-20f9fcae140a-config-volume\") pod \"dns-default-zhmhj\" (UID: \"49f7c8f4-b979-436d-b035-20f9fcae140a\") " pod="openshift-dns/dns-default-zhmhj" Oct 31 00:55:46.725748 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:46.725727 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 00:55:46.725813 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:46.725802 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49f7c8f4-b979-436d-b035-20f9fcae140a-metrics-tls podName:49f7c8f4-b979-436d-b035-20f9fcae140a nodeName:}" failed. No retries permitted until 2026-10-31 00:55:47.225783897 +0000 UTC m=+33.040982758 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49f7c8f4-b979-436d-b035-20f9fcae140a-metrics-tls") pod "dns-default-zhmhj" (UID: "49f7c8f4-b979-436d-b035-20f9fcae140a") : secret "dns-default-metrics-tls" not found Oct 31 00:55:46.725897 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:46.725879 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49f7c8f4-b979-436d-b035-20f9fcae140a-tmp-dir\") pod \"dns-default-zhmhj\" (UID: \"49f7c8f4-b979-436d-b035-20f9fcae140a\") " pod="openshift-dns/dns-default-zhmhj" Oct 31 00:55:46.726125 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:46.726111 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49f7c8f4-b979-436d-b035-20f9fcae140a-config-volume\") pod \"dns-default-zhmhj\" (UID: \"49f7c8f4-b979-436d-b035-20f9fcae140a\") " pod="openshift-dns/dns-default-zhmhj" Oct 31 00:55:46.735107 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:46.735085 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzddr\" (UniqueName: \"kubernetes.io/projected/49f7c8f4-b979-436d-b035-20f9fcae140a-kube-api-access-vzddr\") pod \"dns-default-zhmhj\" (UID: \"49f7c8f4-b979-436d-b035-20f9fcae140a\") " pod="openshift-dns/dns-default-zhmhj" Oct 31 00:55:46.826345 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:46.826304 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnc9j\" (UniqueName: \"kubernetes.io/projected/9a11b541-6b88-439e-bcbe-5901cb4c20b0-kube-api-access-hnc9j\") pod \"ingress-canary-srw8h\" (UID: \"9a11b541-6b88-439e-bcbe-5901cb4c20b0\") " pod="openshift-ingress-canary/ingress-canary-srw8h" Oct 31 00:55:46.826521 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:46.826382 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a11b541-6b88-439e-bcbe-5901cb4c20b0-cert\") pod \"ingress-canary-srw8h\" (UID: \"9a11b541-6b88-439e-bcbe-5901cb4c20b0\") " pod="openshift-ingress-canary/ingress-canary-srw8h" Oct 31 00:55:46.828774 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:46.828756 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a11b541-6b88-439e-bcbe-5901cb4c20b0-cert\") pod \"ingress-canary-srw8h\" (UID: \"9a11b541-6b88-439e-bcbe-5901cb4c20b0\") " pod="openshift-ingress-canary/ingress-canary-srw8h" Oct 31 00:55:46.834310 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:46.834282 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnc9j\" (UniqueName: \"kubernetes.io/projected/9a11b541-6b88-439e-bcbe-5901cb4c20b0-kube-api-access-hnc9j\") pod \"ingress-canary-srw8h\" (UID: \"9a11b541-6b88-439e-bcbe-5901cb4c20b0\") " pod="openshift-ingress-canary/ingress-canary-srw8h" Oct 31 00:55:46.942303 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:46.942272 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-srw8h" Oct 31 00:55:47.229414 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:47.229318 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49f7c8f4-b979-436d-b035-20f9fcae140a-metrics-tls\") pod \"dns-default-zhmhj\" (UID: \"49f7c8f4-b979-436d-b035-20f9fcae140a\") " pod="openshift-dns/dns-default-zhmhj" Oct 31 00:55:47.229566 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:47.229464 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 00:55:47.229566 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:47.229516 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49f7c8f4-b979-436d-b035-20f9fcae140a-metrics-tls podName:49f7c8f4-b979-436d-b035-20f9fcae140a nodeName:}" failed. No retries permitted until 2026-10-31 00:55:48.229502223 +0000 UTC m=+34.044701087 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49f7c8f4-b979-436d-b035-20f9fcae140a-metrics-tls") pod "dns-default-zhmhj" (UID: "49f7c8f4-b979-436d-b035-20f9fcae140a") : secret "dns-default-metrics-tls" not found Oct 31 00:55:47.317206 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:47.317175 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-srw8h"] Oct 31 00:55:47.320589 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:47.320563 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a11b541_6b88_439e_bcbe_5901cb4c20b0.slice/crio-c29ec7474f3f41e866a1a4774fad1dc26dfb897dffc49268023274db226cdeb7 WatchSource:0}: Error finding container c29ec7474f3f41e866a1a4774fad1dc26dfb897dffc49268023274db226cdeb7: Status 404 returned error can't find the container with id c29ec7474f3f41e866a1a4774fad1dc26dfb897dffc49268023274db226cdeb7 Oct 31 00:55:47.454558 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:47.454525 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:47.649384 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:47.649300 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pcvbh" Oct 31 00:55:47.652831 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:47.652801 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 00:55:47.652994 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:47.652961 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-89pfs\"" Oct 31 00:55:47.908201 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:47.908131 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-srw8h" event={"ID":"9a11b541-6b88-439e-bcbe-5901cb4c20b0","Type":"ContainerStarted","Data":"c29ec7474f3f41e866a1a4774fad1dc26dfb897dffc49268023274db226cdeb7"} Oct 31 00:55:48.236974 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:48.236942 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49f7c8f4-b979-436d-b035-20f9fcae140a-metrics-tls\") pod \"dns-default-zhmhj\" (UID: \"49f7c8f4-b979-436d-b035-20f9fcae140a\") " pod="openshift-dns/dns-default-zhmhj" Oct 31 00:55:48.237148 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:48.237082 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 00:55:48.237188 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:48.237161 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49f7c8f4-b979-436d-b035-20f9fcae140a-metrics-tls podName:49f7c8f4-b979-436d-b035-20f9fcae140a nodeName:}" failed. No retries permitted until 2026-10-31 00:55:50.23714105 +0000 UTC m=+36.052339916 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49f7c8f4-b979-436d-b035-20f9fcae140a-metrics-tls") pod "dns-default-zhmhj" (UID: "49f7c8f4-b979-436d-b035-20f9fcae140a") : secret "dns-default-metrics-tls" not found Oct 31 00:55:48.337994 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:48.337957 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/49f8a438-6b29-4160-98e1-3f26e27fe0af-original-pull-secret\") pod \"global-pull-secret-syncer-s9tlh\" (UID: \"49f8a438-6b29-4160-98e1-3f26e27fe0af\") " pod="kube-system/global-pull-secret-syncer-s9tlh" Oct 31 00:55:48.340525 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:48.340498 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/49f8a438-6b29-4160-98e1-3f26e27fe0af-original-pull-secret\") pod \"global-pull-secret-syncer-s9tlh\" (UID: \"49f8a438-6b29-4160-98e1-3f26e27fe0af\") " pod="kube-system/global-pull-secret-syncer-s9tlh" Oct 31 00:55:48.438816 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:48.438777 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f-metrics-certs\") pod \"network-metrics-daemon-pcvbh\" (UID: \"0fe8107c-9128-4c8a-9bde-e1cacd2ed19f\") " pod="openshift-multus/network-metrics-daemon-pcvbh" Oct 31 00:55:48.438957 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:48.438927 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 00:55:48.439003 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:48.438993 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f-metrics-certs podName:0fe8107c-9128-4c8a-9bde-e1cacd2ed19f nodeName:}" failed. No retries permitted until 2026-10-31 00:56:20.438978852 +0000 UTC m=+66.254177703 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f-metrics-certs") pod "network-metrics-daemon-pcvbh" (UID: "0fe8107c-9128-4c8a-9bde-e1cacd2ed19f") : secret "metrics-daemon-secret" not found Oct 31 00:55:48.478529 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:48.478492 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s9tlh" Oct 31 00:55:48.539698 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:48.539668 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwrz5\" (UniqueName: \"kubernetes.io/projected/3c9845ea-a9d6-4d5c-93ea-fb096666a2b9-kube-api-access-kwrz5\") pod \"network-check-target-5xvng\" (UID: \"3c9845ea-a9d6-4d5c-93ea-fb096666a2b9\") " pod="openshift-network-diagnostics/network-check-target-5xvng" Oct 31 00:55:48.543229 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:48.543199 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwrz5\" (UniqueName: \"kubernetes.io/projected/3c9845ea-a9d6-4d5c-93ea-fb096666a2b9-kube-api-access-kwrz5\") pod \"network-check-target-5xvng\" (UID: \"3c9845ea-a9d6-4d5c-93ea-fb096666a2b9\") " pod="openshift-network-diagnostics/network-check-target-5xvng" Oct 31 00:55:48.797609 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:48.797523 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-s9tlh"] Oct 31 00:55:48.799602 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:48.799585 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xvng" Oct 31 00:55:48.801070 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:48.801031 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49f8a438_6b29_4160_98e1_3f26e27fe0af.slice/crio-1a780d8ff00f4f054342bb3d9a6fa482cc171917eacbf946303c50dc465eebfe WatchSource:0}: Error finding container 1a780d8ff00f4f054342bb3d9a6fa482cc171917eacbf946303c50dc465eebfe: Status 404 returned error can't find the container with id 1a780d8ff00f4f054342bb3d9a6fa482cc171917eacbf946303c50dc465eebfe Oct 31 00:55:48.849949 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:48.849884 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:48.912142 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:48.912108 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-s9tlh" event={"ID":"49f8a438-6b29-4160-98e1-3f26e27fe0af","Type":"ContainerStarted","Data":"1a780d8ff00f4f054342bb3d9a6fa482cc171917eacbf946303c50dc465eebfe"} Oct 31 00:55:48.950578 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:48.950543 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:49.104038 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:49.104007 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5xvng"] Oct 31 00:55:49.109174 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:49.109147 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c9845ea_a9d6_4d5c_93ea_fb096666a2b9.slice/crio-461b4f157fce847de1f83cb10e8a77c9a2c95b98de007c48546443716ebefb36 WatchSource:0}: Error finding container 461b4f157fce847de1f83cb10e8a77c9a2c95b98de007c48546443716ebefb36: Status 404 returned error can't find the container with id 461b4f157fce847de1f83cb10e8a77c9a2c95b98de007c48546443716ebefb36 Oct 31 00:55:49.916576 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:49.916494 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5xvng" event={"ID":"3c9845ea-a9d6-4d5c-93ea-fb096666a2b9","Type":"ContainerStarted","Data":"461b4f157fce847de1f83cb10e8a77c9a2c95b98de007c48546443716ebefb36"} Oct 31 00:55:50.253102 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:50.253057 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49f7c8f4-b979-436d-b035-20f9fcae140a-metrics-tls\") pod \"dns-default-zhmhj\" (UID: \"49f7c8f4-b979-436d-b035-20f9fcae140a\") " pod="openshift-dns/dns-default-zhmhj" Oct 31 00:55:50.253299 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:50.253263 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 00:55:50.253368 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:50.253335 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49f7c8f4-b979-436d-b035-20f9fcae140a-metrics-tls podName:49f7c8f4-b979-436d-b035-20f9fcae140a nodeName:}" failed. No retries permitted until 2026-10-31 00:55:54.25331618 +0000 UTC m=+40.068515034 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49f7c8f4-b979-436d-b035-20f9fcae140a-metrics-tls") pod "dns-default-zhmhj" (UID: "49f7c8f4-b979-436d-b035-20f9fcae140a") : secret "dns-default-metrics-tls" not found Oct 31 00:55:50.398042 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:50.397132 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-lb9mz"] Oct 31 00:55:50.416627 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:50.415756 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lb9mz" Oct 31 00:55:50.418609 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:50.418537 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-lb9mz"] Oct 31 00:55:50.422891 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:50.422863 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-ht5gb\"" Oct 31 00:55:50.423863 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:50.423801 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:50.424142 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:50.424095 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 31 00:55:50.424142 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:50.424095 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 31 00:55:50.524673 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:50.524579 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:50.556284 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:50.556243 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfhcf\" (UniqueName: \"kubernetes.io/projected/ac0f8a44-5a66-40fe-a08c-080314e7ea60-kube-api-access-hfhcf\") pod \"migrator-66df5d496b-lb9mz\" (UID: \"ac0f8a44-5a66-40fe-a08c-080314e7ea60\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lb9mz" Oct 31 00:55:50.647900 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:50.647867 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:50.657100 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:50.657057 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfhcf\" (UniqueName: \"kubernetes.io/projected/ac0f8a44-5a66-40fe-a08c-080314e7ea60-kube-api-access-hfhcf\") pod \"migrator-66df5d496b-lb9mz\" (UID: \"ac0f8a44-5a66-40fe-a08c-080314e7ea60\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lb9mz" Oct 31 00:55:50.670287 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:50.670254 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfhcf\" (UniqueName: \"kubernetes.io/projected/ac0f8a44-5a66-40fe-a08c-080314e7ea60-kube-api-access-hfhcf\") pod \"migrator-66df5d496b-lb9mz\" (UID: \"ac0f8a44-5a66-40fe-a08c-080314e7ea60\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lb9mz" Oct 31 00:55:50.758056 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:50.758016 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lb9mz" Oct 31 00:55:50.768050 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:50.767964 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:51.056931 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:51.056857 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-lb9mz"] Oct 31 00:55:51.071004 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:51.068777 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac0f8a44_5a66_40fe_a08c_080314e7ea60.slice/crio-e3d7e294ddd6e2bc291ade2bc81bae50733485bf3303acfd8fe2d4c137845eaa WatchSource:0}: Error finding container e3d7e294ddd6e2bc291ade2bc81bae50733485bf3303acfd8fe2d4c137845eaa: Status 404 returned error can't find the container with id e3d7e294ddd6e2bc291ade2bc81bae50733485bf3303acfd8fe2d4c137845eaa Oct 31 00:55:51.781301 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:51.781263 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:51.893495 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:51.892590 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:51.910261 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:51.910231 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-sk9vt"] Oct 31 00:55:51.935414 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:51.935384 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-sk9vt"] Oct 31 00:55:51.935582 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:51.935448 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lb9mz" event={"ID":"ac0f8a44-5a66-40fe-a08c-080314e7ea60","Type":"ContainerStarted","Data":"e3d7e294ddd6e2bc291ade2bc81bae50733485bf3303acfd8fe2d4c137845eaa"} Oct 31 00:55:51.935582 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:51.935554 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-sk9vt" Oct 31 00:55:51.939981 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:51.939945 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 31 00:55:51.941118 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:51.941092 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-xg6v4\"" Oct 31 00:55:51.944569 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:51.944517 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 31 00:55:51.944569 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:51.944533 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 31 00:55:51.944737 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:51.944675 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 31 00:55:52.003372 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:52.003336 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:52.069826 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:52.069687 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klvmv\" (UniqueName: \"kubernetes.io/projected/ee8bc72c-f181-452d-ab71-b13a51672bcf-kube-api-access-klvmv\") pod \"service-ca-5fffc97bdf-sk9vt\" (UID: \"ee8bc72c-f181-452d-ab71-b13a51672bcf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sk9vt" Oct 31 00:55:52.069826 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:52.069729 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ee8bc72c-f181-452d-ab71-b13a51672bcf-signing-cabundle\") pod \"service-ca-5fffc97bdf-sk9vt\" (UID: \"ee8bc72c-f181-452d-ab71-b13a51672bcf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sk9vt" Oct 31 00:55:52.070346 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:52.069885 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ee8bc72c-f181-452d-ab71-b13a51672bcf-signing-key\") pod \"service-ca-5fffc97bdf-sk9vt\" (UID: \"ee8bc72c-f181-452d-ab71-b13a51672bcf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sk9vt" Oct 31 00:55:52.170967 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:52.170928 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ee8bc72c-f181-452d-ab71-b13a51672bcf-signing-key\") pod \"service-ca-5fffc97bdf-sk9vt\" (UID: \"ee8bc72c-f181-452d-ab71-b13a51672bcf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sk9vt" Oct 31 00:55:52.171143 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:52.170988 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klvmv\" (UniqueName: \"kubernetes.io/projected/ee8bc72c-f181-452d-ab71-b13a51672bcf-kube-api-access-klvmv\") pod \"service-ca-5fffc97bdf-sk9vt\" (UID: \"ee8bc72c-f181-452d-ab71-b13a51672bcf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sk9vt" Oct 31 00:55:52.171143 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:52.171032 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ee8bc72c-f181-452d-ab71-b13a51672bcf-signing-cabundle\") pod \"service-ca-5fffc97bdf-sk9vt\" (UID: \"ee8bc72c-f181-452d-ab71-b13a51672bcf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sk9vt" Oct 31 00:55:52.171960 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:52.171929 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ee8bc72c-f181-452d-ab71-b13a51672bcf-signing-cabundle\") pod \"service-ca-5fffc97bdf-sk9vt\" (UID: \"ee8bc72c-f181-452d-ab71-b13a51672bcf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sk9vt" Oct 31 00:55:52.174277 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:52.174247 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ee8bc72c-f181-452d-ab71-b13a51672bcf-signing-key\") pod \"service-ca-5fffc97bdf-sk9vt\" (UID: \"ee8bc72c-f181-452d-ab71-b13a51672bcf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sk9vt" Oct 31 00:55:52.179662 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:52.179619 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klvmv\" (UniqueName: \"kubernetes.io/projected/ee8bc72c-f181-452d-ab71-b13a51672bcf-kube-api-access-klvmv\") pod \"service-ca-5fffc97bdf-sk9vt\" (UID: \"ee8bc72c-f181-452d-ab71-b13a51672bcf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sk9vt" Oct 31 00:55:52.260348 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:52.260311 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-sk9vt" Oct 31 00:55:52.562234 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:52.562195 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-sk9vt"] Oct 31 00:55:52.657100 ip-10-0-130-39 kubenswrapper[2576]: W1031 00:55:52.657019 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee8bc72c_f181_452d_ab71_b13a51672bcf.slice/crio-a83111a61a2f22ec5001f9fa3036f6a7bd673feb30fc55e65ea40f609e2a974e WatchSource:0}: Error finding container a83111a61a2f22ec5001f9fa3036f6a7bd673feb30fc55e65ea40f609e2a974e: Status 404 returned error can't find the container with id a83111a61a2f22ec5001f9fa3036f6a7bd673feb30fc55e65ea40f609e2a974e Oct 31 00:55:52.850590 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:52.850562 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:52.928657 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:52.928621 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-srw8h" event={"ID":"9a11b541-6b88-439e-bcbe-5901cb4c20b0","Type":"ContainerStarted","Data":"769d5aa6160f164bcbb357690d8d0bd2d4f523e5900c29124e36e6f373669f1c"} Oct 31 00:55:52.930058 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:52.930028 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-sk9vt" event={"ID":"ee8bc72c-f181-452d-ab71-b13a51672bcf","Type":"ContainerStarted","Data":"a83111a61a2f22ec5001f9fa3036f6a7bd673feb30fc55e65ea40f609e2a974e"} Oct 31 00:55:52.931301 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:52.931278 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lb9mz" event={"ID":"ac0f8a44-5a66-40fe-a08c-080314e7ea60","Type":"ContainerStarted","Data":"05ee59201ddbfac3487124f860a2bf316e85f4716910b493df9d34039639e412"} Oct 31 00:55:52.952202 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:52.952155 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-srw8h" podStartSLOduration=2.404578878 podStartE2EDuration="6.952143089s" totalImagesPullingTime="4.547564211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:55:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:52.950618839 +0000 UTC m=+38.765817711" watchObservedRunningTime="2026-10-31 00:55:52.952143089 +0000 UTC m=+38.767341953" Oct 31 00:55:52.971710 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:52.971670 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lb9mz" podStartSLOduration=1.3667090179999999 podStartE2EDuration="2.971657852s" totalImagesPullingTime="1.604948834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:55:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:52.97124958 +0000 UTC m=+38.786448468" watchObservedRunningTime="2026-10-31 00:55:52.971657852 +0000 UTC m=+38.786856719" Oct 31 00:55:53.937784 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:53.937678 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5xvng" event={"ID":"3c9845ea-a9d6-4d5c-93ea-fb096666a2b9","Type":"ContainerStarted","Data":"4facd1e9617951d6a484a99146b70bae42adb0a192883d7cff4302799cf08072"} Oct 31 00:55:53.962259 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:53.962194 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5xvng" podStartSLOduration=34.960526568 podStartE2EDuration="38.962178174s" totalImagesPullingTime="4.001651606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:55:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:53.961412071 +0000 UTC m=+39.776610947" watchObservedRunningTime="2026-10-31 00:55:53.962178174 +0000 UTC m=+39.777377046" Oct 31 00:55:54.286742 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:54.286653 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49f7c8f4-b979-436d-b035-20f9fcae140a-metrics-tls\") pod \"dns-default-zhmhj\" (UID: \"49f7c8f4-b979-436d-b035-20f9fcae140a\") " pod="openshift-dns/dns-default-zhmhj" Oct 31 00:55:54.286916 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:54.286796 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 00:55:54.286916 ip-10-0-130-39 kubenswrapper[2576]: E1031 00:55:54.286870 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49f7c8f4-b979-436d-b035-20f9fcae140a-metrics-tls podName:49f7c8f4-b979-436d-b035-20f9fcae140a nodeName:}" failed. No retries permitted until 2026-10-31 00:56:02.286852098 +0000 UTC m=+48.102050949 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49f7c8f4-b979-436d-b035-20f9fcae140a-metrics-tls") pod "dns-default-zhmhj" (UID: "49f7c8f4-b979-436d-b035-20f9fcae140a") : secret "dns-default-metrics-tls" not found Oct 31 00:55:54.940353 ip-10-0-130-39 kubenswrapper[2576]: I1031 00:55:54.940323 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5xvng" Oct 31 00:55:55.570437 ip-10-0-130-39 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 00:55:55.585900 ip-10-0-130-39 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 00:55:55.586101 ip-10-0-130-39 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 00:55:55.586163 ip-10-0-130-39 systemd[1]: kubelet.service: Consumed 2.380s CPU time. Oct 31 00:55:55.609960 ip-10-0-130-39 systemd[1]: Starting Kubernetes Kubelet... Oct 31 00:55:55.682940 ip-10-0-130-39 kubenswrapper[4440]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:55:55.682940 ip-10-0-130-39 kubenswrapper[4440]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 00:55:55.682940 ip-10-0-130-39 kubenswrapper[4440]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:55:55.682940 ip-10-0-130-39 kubenswrapper[4440]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:55:55.685129 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685040 4440 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:55:55.685129 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685129 4440 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:55:55.685129 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685134 4440 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:55:55.685237 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685138 4440 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:55:55.685237 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685141 4440 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:55:55.685237 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685144 4440 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:55:55.685237 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685147 4440 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:55:55.685237 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685151 4440 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:55:55.685237 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685153 4440 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:55:55.685237 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685156 4440 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:55:55.685237 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685159 4440 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:55:55.685237 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685162 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:55:55.685237 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685165 4440 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:55:55.685237 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685167 4440 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:55:55.685237 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685170 4440 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:55:55.685237 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685173 4440 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:55:55.685237 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685175 4440 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:55:55.685237 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685178 4440 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:55:55.685237 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685180 4440 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:55:55.685237 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685183 4440 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:55:55.685237 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685187 4440 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:55:55.685237 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685190 4440 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:55:55.685237 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685192 4440 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:55:55.685750 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685195 4440 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:55:55.685750 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685198 4440 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:55:55.685750 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685200 4440 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:55:55.685750 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685211 4440 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:55:55.685750 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685214 4440 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:55:55.685750 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685216 4440 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:55:55.685750 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685221 4440 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:55:55.685750 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685226 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:55:55.685750 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685229 4440 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:55:55.685750 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685232 4440 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:55:55.685750 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685235 4440 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:55:55.685750 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685237 4440 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:55:55.685750 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685240 4440 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:55:55.685750 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685243 4440 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:55:55.685750 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685246 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:55:55.685750 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685249 4440 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:55:55.685750 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685252 4440 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:55:55.685750 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685255 4440 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:55:55.685750 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685257 4440 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:55:55.686263 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685260 4440 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:55:55.686263 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685263 4440 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:55:55.686263 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685266 4440 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:55:55.686263 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685268 4440 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:55:55.686263 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685271 4440 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:55:55.686263 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685273 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:55:55.686263 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685276 4440 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:55:55.686263 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685278 4440 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:55:55.686263 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685281 4440 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:55:55.686263 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685283 4440 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:55:55.686263 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685286 4440 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:55:55.686263 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685289 4440 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:55:55.686263 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685291 4440 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:55:55.686263 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685294 4440 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:55:55.686263 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685297 4440 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:55:55.686263 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685300 4440 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:55:55.686263 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685303 4440 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:55:55.686263 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685306 4440 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:55:55.686263 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685309 4440 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:55:55.686263 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685311 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:55:55.686778 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685315 4440 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:55:55.686778 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685321 4440 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:55:55.686778 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685324 4440 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:55:55.686778 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685327 4440 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:55:55.686778 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685329 4440 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:55:55.686778 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685332 4440 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:55:55.686778 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685334 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:55:55.686778 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685337 4440 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:55:55.686778 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685340 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:55:55.686778 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685342 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:55:55.686778 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685345 4440 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:55:55.686778 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685348 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:55:55.686778 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685350 4440 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:55:55.686778 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685353 4440 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:55:55.686778 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685355 4440 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:55:55.686778 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685357 4440 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:55:55.686778 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685360 4440 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:55:55.686778 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685362 4440 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:55:55.686778 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685365 4440 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:55:55.687262 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685367 4440 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:55:55.687262 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685370 4440 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:55:55.687262 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685372 4440 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:55:55.687262 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685375 4440 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:55:55.687262 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685379 4440 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:55:55.687262 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685381 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:55:55.687262 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685384 4440 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:55:55.687262 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685387 4440 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:55:55.687262 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685390 4440 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:55:55.687262 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685393 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:55:55.687262 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685395 4440 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:55:55.687262 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685398 4440 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:55:55.687262 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685401 4440 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:55:55.687262 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685403 4440 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:55:55.687262 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685405 4440 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:55:55.687262 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685408 4440 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:55:55.687262 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685411 4440 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:55:55.687262 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685413 4440 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:55:55.687262 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685439 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:55:55.687746 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685444 4440 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:55:55.687746 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685448 4440 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:55:55.687746 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685452 4440 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:55:55.687746 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685456 4440 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:55:55.687746 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685462 4440 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:55:55.687746 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.685466 4440 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:55:55.687746 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686297 4440 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:55:55.687746 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686306 4440 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:55:55.687746 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686309 4440 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:55:55.687746 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686313 4440 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:55:55.687746 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686316 4440 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:55:55.687746 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686318 4440 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:55:55.687746 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686321 4440 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:55:55.687746 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686324 4440 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:55:55.687746 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686327 4440 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:55:55.687746 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686329 4440 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:55:55.687746 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686334 4440 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:55:55.687746 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686337 4440 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:55:55.687746 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686341 4440 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:55:55.688221 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686344 4440 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:55:55.688221 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686349 4440 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:55:55.688221 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686352 4440 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:55:55.688221 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686355 4440 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:55:55.688221 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686358 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:55:55.688221 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686360 4440 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:55:55.688221 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686363 4440 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:55:55.688221 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686365 4440 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:55:55.688221 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686368 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:55:55.688221 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686371 4440 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:55:55.688221 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686374 4440 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:55:55.688221 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686376 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:55:55.688221 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686379 4440 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:55:55.688221 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686381 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:55:55.688221 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686384 4440 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:55:55.688221 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686387 4440 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:55:55.688221 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686389 4440 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:55:55.688221 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686392 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:55:55.688221 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686394 4440 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:55:55.688710 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686397 4440 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:55:55.688710 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686400 4440 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:55:55.688710 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686403 4440 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:55:55.688710 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686405 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:55:55.688710 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686408 4440 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:55:55.688710 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686411 4440 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:55:55.688710 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686413 4440 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:55:55.688710 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686430 4440 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:55:55.688710 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686432 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:55:55.688710 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686435 4440 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:55:55.688710 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686438 4440 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:55:55.688710 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686440 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:55:55.688710 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686448 4440 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:55:55.688710 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686454 4440 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:55:55.688710 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686458 4440 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:55:55.688710 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686461 4440 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:55:55.688710 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686463 4440 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:55:55.688710 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686466 4440 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:55:55.688710 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686469 4440 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:55:55.688710 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686471 4440 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:55:55.689209 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686474 4440 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:55:55.689209 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686477 4440 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:55:55.689209 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686480 4440 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:55:55.689209 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686482 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:55:55.689209 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686485 4440 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:55:55.689209 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686487 4440 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:55:55.689209 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686490 4440 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:55:55.689209 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686493 4440 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:55:55.689209 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686495 4440 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:55:55.689209 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686498 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:55:55.689209 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686500 4440 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:55:55.689209 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686503 4440 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:55:55.689209 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686505 4440 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:55:55.689209 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686508 4440 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:55:55.689209 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686510 4440 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:55:55.689209 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686514 4440 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:55:55.689209 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686516 4440 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:55:55.689209 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686519 4440 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:55:55.689209 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686522 4440 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:55:55.689209 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686524 4440 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:55:55.689719 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686527 4440 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:55:55.689719 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686529 4440 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:55:55.689719 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686532 4440 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:55:55.689719 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686535 4440 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:55:55.689719 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686539 4440 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:55:55.689719 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686543 4440 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:55:55.689719 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686553 4440 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:55:55.689719 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686556 4440 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:55:55.689719 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686559 4440 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:55:55.689719 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686561 4440 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:55:55.689719 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686564 4440 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:55:55.689719 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686566 4440 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:55:55.689719 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686569 4440 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:55:55.689719 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686572 4440 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:55:55.689719 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686574 4440 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:55:55.689719 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686577 4440 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:55:55.689719 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686579 4440 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:55:55.689719 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686582 4440 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:55:55.689719 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686584 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:55:55.689719 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686587 4440 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:55:55.690225 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686589 4440 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:55:55.690225 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686592 4440 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:55:55.690225 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686594 4440 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:55:55.690225 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686597 4440 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:55:55.690225 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686599 4440 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:55:55.690225 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686602 4440 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:55:55.690225 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686605 4440 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:55:55.690225 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686608 4440 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:55:55.690225 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686611 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:55:55.690225 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686614 4440 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:55:55.690225 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686616 4440 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:55:55.690225 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686619 4440 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:55:55.690225 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686621 4440 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:55:55.690225 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.686623 4440 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:55:55.690225 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687157 4440 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:55:55.690225 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687162 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:55:55.690225 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687168 4440 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:55:55.690225 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687170 4440 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:55:55.690225 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687175 4440 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:55:55.690700 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687178 4440 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:55:55.690700 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687181 4440 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:55:55.690700 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687184 4440 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:55:55.690700 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687186 4440 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:55:55.690700 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687189 4440 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:55:55.690700 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687191 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:55:55.690700 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687194 4440 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:55:55.690700 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687197 4440 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:55:55.690700 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687199 4440 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:55:55.690700 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687202 4440 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:55:55.690700 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687204 4440 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:55:55.690700 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687207 4440 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:55:55.690700 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687210 4440 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:55:55.690700 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687212 4440 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:55:55.690700 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687215 4440 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:55:55.690700 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687217 4440 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:55:55.690700 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687220 4440 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:55:55.690700 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687223 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:55:55.690700 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687225 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:55:55.690700 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687228 4440 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:55:55.691279 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687231 4440 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:55:55.691279 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687233 4440 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:55:55.691279 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687236 4440 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:55:55.691279 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687238 4440 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:55:55.691279 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687241 4440 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:55:55.691279 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687243 4440 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:55:55.691279 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687246 4440 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:55:55.691279 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687248 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:55:55.691279 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687251 4440 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:55:55.691279 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687254 4440 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:55:55.691279 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687257 4440 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:55:55.691279 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687260 4440 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:55:55.691279 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687264 4440 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:55:55.691279 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687267 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:55:55.691279 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687270 4440 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:55:55.691279 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687273 4440 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:55:55.691279 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687277 4440 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:55:55.691279 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687281 4440 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:55:55.691279 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687283 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:55:55.692143 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687286 4440 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:55:55.692143 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687289 4440 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:55:55.692143 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687291 4440 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:55:55.692143 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687294 4440 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:55:55.692143 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687297 4440 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:55:55.692143 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687299 4440 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:55:55.692143 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687302 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:55:55.692143 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687304 4440 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:55:55.692143 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687307 4440 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:55:55.692143 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687309 4440 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:55:55.692143 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687312 4440 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:55:55.692143 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687315 4440 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:55:55.692143 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687317 4440 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:55:55.692143 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687320 4440 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:55:55.692143 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687322 4440 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:55:55.692143 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687325 4440 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:55:55.692143 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687327 4440 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:55:55.692143 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687330 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:55:55.692143 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687332 4440 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:55:55.693005 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687335 4440 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:55:55.693005 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687338 4440 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:55:55.693005 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687341 4440 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:55:55.693005 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687344 4440 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:55:55.693005 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687347 4440 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:55:55.693005 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687350 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:55:55.693005 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687353 4440 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:55:55.693005 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687356 4440 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:55:55.693005 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687358 4440 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:55:55.693005 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687361 4440 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:55:55.693005 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687364 4440 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:55:55.693005 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687367 4440 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:55:55.693005 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687371 4440 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:55:55.693005 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687374 4440 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:55:55.693005 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687376 4440 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:55:55.693005 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687380 4440 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:55:55.693005 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687384 4440 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:55:55.693005 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687387 4440 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:55:55.693005 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687389 4440 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:55:55.693005 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687392 4440 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:55:55.693793 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687394 4440 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:55:55.693793 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687397 4440 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:55:55.693793 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687400 4440 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:55:55.693793 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687402 4440 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:55:55.693793 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687405 4440 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:55:55.693793 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687408 4440 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:55:55.693793 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687410 4440 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:55:55.693793 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687413 4440 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:55:55.693793 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687428 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:55:55.693793 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687431 4440 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:55:55.693793 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687433 4440 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:55:55.693793 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687436 4440 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:55:55.693793 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687439 4440 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:55:55.693793 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687442 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:55:55.693793 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687445 4440 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:55:55.693793 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687449 4440 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:55:55.693793 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687451 4440 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:55:55.693793 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687454 4440 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:55:55.693793 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687457 4440 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:55:55.694651 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687459 4440 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:55:55.694651 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687462 4440 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:55:55.694651 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687465 4440 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:55:55.694651 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.687468 4440 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:55:55.694651 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687565 4440 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 00:55:55.694651 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687576 4440 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 00:55:55.694651 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687582 4440 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 00:55:55.694651 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687587 4440 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 00:55:55.694651 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687591 4440 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 00:55:55.694651 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687594 4440 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 00:55:55.694651 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687599 4440 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 00:55:55.694651 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687603 4440 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 00:55:55.694651 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687607 4440 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 00:55:55.694651 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687610 4440 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 00:55:55.694651 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687613 4440 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 00:55:55.694651 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687617 4440 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 00:55:55.694651 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687621 4440 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 00:55:55.694651 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687624 4440 flags.go:64] FLAG: --cgroup-root="" Oct 31 00:55:55.694651 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687627 4440 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 00:55:55.694651 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687631 4440 flags.go:64] FLAG: --client-ca-file="" Oct 31 00:55:55.694651 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687634 4440 flags.go:64] FLAG: --cloud-provider="external" Oct 31 00:55:55.694651 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687637 4440 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 00:55:55.694651 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687642 4440 flags.go:64] FLAG: --cluster-domain="" Oct 31 00:55:55.694651 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687645 4440 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 00:55:55.695712 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687649 4440 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 00:55:55.695712 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687652 4440 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 00:55:55.695712 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687656 4440 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 00:55:55.695712 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687660 4440 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 00:55:55.695712 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687664 4440 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 00:55:55.695712 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687668 4440 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 00:55:55.695712 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687671 4440 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 00:55:55.695712 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687674 4440 flags.go:64] FLAG: --contention-profiling="false" Oct 31 00:55:55.695712 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687677 4440 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 00:55:55.695712 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687680 4440 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 00:55:55.695712 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687684 4440 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 00:55:55.695712 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687687 4440 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 00:55:55.695712 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687692 4440 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 00:55:55.695712 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687697 4440 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 00:55:55.695712 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687699 4440 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 00:55:55.695712 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687702 4440 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 00:55:55.695712 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687706 4440 flags.go:64] FLAG: --enable-server="true" Oct 31 00:55:55.695712 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687709 4440 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 00:55:55.695712 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687713 4440 flags.go:64] FLAG: --event-burst="100" Oct 31 00:55:55.695712 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687717 4440 flags.go:64] FLAG: --event-qps="50" Oct 31 00:55:55.695712 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687720 4440 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 00:55:55.695712 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687723 4440 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 00:55:55.695712 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687726 4440 flags.go:64] FLAG: --eviction-hard="" Oct 31 00:55:55.695712 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687730 4440 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 00:55:55.696801 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687733 4440 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 00:55:55.696801 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687736 4440 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 00:55:55.696801 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687740 4440 flags.go:64] FLAG: --eviction-soft="" Oct 31 00:55:55.696801 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687743 4440 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 00:55:55.696801 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687746 4440 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 00:55:55.696801 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687749 4440 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 00:55:55.696801 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687752 4440 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 00:55:55.696801 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687755 4440 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 00:55:55.696801 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687758 4440 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 00:55:55.696801 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687761 4440 flags.go:64] FLAG: --feature-gates="" Oct 31 00:55:55.696801 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687765 4440 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 00:55:55.696801 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687769 4440 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 00:55:55.696801 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687773 4440 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 00:55:55.696801 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687777 4440 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 00:55:55.696801 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687780 4440 flags.go:64] FLAG: --healthz-port="10248" Oct 31 00:55:55.696801 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687783 4440 flags.go:64] FLAG: --help="false" Oct 31 00:55:55.696801 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687786 4440 flags.go:64] FLAG: --hostname-override="ip-10-0-130-39.us-east-2.compute.internal" Oct 31 00:55:55.696801 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687789 4440 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 00:55:55.696801 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687793 4440 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 00:55:55.696801 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687797 4440 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 00:55:55.696801 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687800 4440 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 00:55:55.696801 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687805 4440 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 00:55:55.696801 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687808 4440 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 00:55:55.697844 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687812 4440 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 00:55:55.697844 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687814 4440 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 00:55:55.697844 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687817 4440 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 00:55:55.697844 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687820 4440 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 00:55:55.697844 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687824 4440 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 00:55:55.697844 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687827 4440 flags.go:64] FLAG: --kube-reserved="" Oct 31 00:55:55.697844 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687830 4440 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 00:55:55.697844 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687832 4440 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 00:55:55.697844 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687835 4440 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 00:55:55.697844 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687838 4440 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 00:55:55.697844 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687841 4440 flags.go:64] FLAG: --lock-file="" Oct 31 00:55:55.697844 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687844 4440 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 00:55:55.697844 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687847 4440 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 00:55:55.697844 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687850 4440 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 00:55:55.697844 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687856 4440 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 00:55:55.697844 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687859 4440 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 00:55:55.697844 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687862 4440 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 00:55:55.697844 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687865 4440 flags.go:64] FLAG: --logging-format="text" Oct 31 00:55:55.697844 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687868 4440 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 00:55:55.697844 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687871 4440 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 00:55:55.697844 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687874 4440 flags.go:64] FLAG: --manifest-url="" Oct 31 00:55:55.697844 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687878 4440 flags.go:64] FLAG: --manifest-url-header="" Oct 31 00:55:55.697844 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687883 4440 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 00:55:55.697844 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687886 4440 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 00:55:55.697844 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687890 4440 flags.go:64] FLAG: --max-pods="110" Oct 31 00:55:55.699065 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687893 4440 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 00:55:55.699065 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687896 4440 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 00:55:55.699065 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687899 4440 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 00:55:55.699065 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687902 4440 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 00:55:55.699065 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687906 4440 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 00:55:55.699065 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687910 4440 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 00:55:55.699065 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687914 4440 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 00:55:55.699065 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687921 4440 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 00:55:55.699065 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687924 4440 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 00:55:55.699065 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687927 4440 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 00:55:55.699065 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687930 4440 flags.go:64] FLAG: --pod-cidr="" Oct 31 00:55:55.699065 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687933 4440 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 00:55:55.699065 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687936 4440 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 00:55:55.699065 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687939 4440 flags.go:64] FLAG: --pods-per-core="0" Oct 31 00:55:55.699065 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687942 4440 flags.go:64] FLAG: --port="10250" Oct 31 00:55:55.699065 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687945 4440 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 00:55:55.699065 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687948 4440 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0dd13a34fbb3e16a9" Oct 31 00:55:55.699065 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687951 4440 flags.go:64] FLAG: --qos-reserved="" Oct 31 00:55:55.699065 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687955 4440 flags.go:64] FLAG: --read-only-port="10255" Oct 31 00:55:55.699065 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687958 4440 flags.go:64] FLAG: --register-node="true" Oct 31 00:55:55.699065 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687960 4440 flags.go:64] FLAG: --register-with-taints="" Oct 31 00:55:55.699065 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687967 4440 flags.go:64] FLAG: --registry-burst="10" Oct 31 00:55:55.699065 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687970 4440 flags.go:64] FLAG: --registry-qps="5" Oct 31 00:55:55.699065 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687973 4440 flags.go:64] FLAG: --reserved-cpus="" Oct 31 00:55:55.699065 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687976 4440 flags.go:64] FLAG: --reserved-memory="" Oct 31 00:55:55.699065 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687981 4440 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 00:55:55.700321 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687984 4440 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 00:55:55.700321 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687987 4440 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 00:55:55.700321 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687992 4440 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 00:55:55.700321 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687995 4440 flags.go:64] FLAG: --runonce="false" Oct 31 00:55:55.700321 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.687998 4440 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 00:55:55.700321 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.688001 4440 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 00:55:55.700321 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.688004 4440 flags.go:64] FLAG: --seccomp-default="false" Oct 31 00:55:55.700321 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.688007 4440 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 00:55:55.700321 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.688010 4440 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 00:55:55.700321 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.688013 4440 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 00:55:55.700321 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.688017 4440 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 00:55:55.700321 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.688025 4440 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 00:55:55.700321 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.688029 4440 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 00:55:55.700321 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.688032 4440 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 00:55:55.700321 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.688035 4440 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 00:55:55.700321 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.688038 4440 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 00:55:55.700321 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.688041 4440 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 00:55:55.700321 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.688044 4440 flags.go:64] FLAG: --system-cgroups="" Oct 31 00:55:55.700321 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.688047 4440 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 00:55:55.700321 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.688052 4440 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 00:55:55.700321 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.688055 4440 flags.go:64] FLAG: --tls-cert-file="" Oct 31 00:55:55.700321 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.688058 4440 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 00:55:55.700321 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.688063 4440 flags.go:64] FLAG: --tls-min-version="" Oct 31 00:55:55.700321 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.688066 4440 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 00:55:55.700321 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.688069 4440 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 00:55:55.701449 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.688072 4440 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 00:55:55.701449 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.688075 4440 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 00:55:55.701449 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.688078 4440 flags.go:64] FLAG: --v="2" Oct 31 00:55:55.701449 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.688083 4440 flags.go:64] FLAG: --version="false" Oct 31 00:55:55.701449 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.688087 4440 flags.go:64] FLAG: --vmodule="" Oct 31 00:55:55.701449 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.688091 4440 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 00:55:55.701449 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.688094 4440 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 00:55:55.701449 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688230 4440 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:55:55.701449 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688233 4440 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:55:55.701449 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688238 4440 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:55:55.701449 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688240 4440 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:55:55.701449 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688243 4440 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:55:55.701449 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688246 4440 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:55:55.701449 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688249 4440 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:55:55.701449 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688252 4440 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:55:55.701449 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688256 4440 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:55:55.701449 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688261 4440 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:55:55.701449 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688265 4440 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:55:55.701449 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688270 4440 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:55:55.701449 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688274 4440 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:55:55.701449 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688277 4440 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:55:55.702328 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688280 4440 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:55:55.702328 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688283 4440 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:55:55.702328 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688286 4440 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:55:55.702328 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688289 4440 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:55:55.702328 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688291 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:55:55.702328 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688294 4440 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:55:55.702328 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688296 4440 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:55:55.702328 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688299 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:55:55.702328 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688302 4440 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:55:55.702328 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688304 4440 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:55:55.702328 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688307 4440 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:55:55.702328 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688310 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:55:55.702328 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688312 4440 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:55:55.702328 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688315 4440 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:55:55.702328 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688318 4440 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:55:55.702328 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688320 4440 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:55:55.702328 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688323 4440 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:55:55.702328 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688326 4440 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:55:55.702328 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688328 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:55:55.703079 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688331 4440 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:55:55.703079 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688334 4440 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:55:55.703079 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688336 4440 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:55:55.703079 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688339 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:55:55.703079 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688342 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:55:55.703079 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688344 4440 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:55:55.703079 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688347 4440 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:55:55.703079 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688349 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:55:55.703079 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688352 4440 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:55:55.703079 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688354 4440 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:55:55.703079 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688359 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:55:55.703079 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688361 4440 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:55:55.703079 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688364 4440 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:55:55.703079 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688367 4440 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:55:55.703079 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688370 4440 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:55:55.703079 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688373 4440 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:55:55.703079 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688375 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:55:55.703079 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688378 4440 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:55:55.703079 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688380 4440 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:55:55.703079 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688382 4440 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:55:55.703696 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688385 4440 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:55:55.703696 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688387 4440 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:55:55.703696 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688390 4440 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:55:55.703696 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688393 4440 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:55:55.703696 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688395 4440 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:55:55.703696 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688398 4440 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:55:55.703696 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688401 4440 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:55:55.703696 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688403 4440 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:55:55.703696 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688406 4440 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:55:55.703696 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688409 4440 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:55:55.703696 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688411 4440 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:55:55.703696 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688428 4440 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:55:55.703696 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688431 4440 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:55:55.703696 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688434 4440 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:55:55.703696 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688437 4440 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:55:55.703696 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688439 4440 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:55:55.703696 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688442 4440 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:55:55.703696 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688444 4440 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:55:55.703696 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688447 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:55:55.703696 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688449 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:55:55.704356 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688452 4440 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:55:55.704356 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688455 4440 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:55:55.704356 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688460 4440 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:55:55.704356 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688463 4440 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:55:55.704356 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688466 4440 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:55:55.704356 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688469 4440 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:55:55.704356 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688472 4440 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:55:55.704356 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688475 4440 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:55:55.704356 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688477 4440 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:55:55.704356 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688480 4440 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:55:55.704356 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688483 4440 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:55:55.704356 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688485 4440 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:55:55.704356 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688488 4440 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:55:55.704356 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688491 4440 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:55:55.704356 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688493 4440 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:55:55.704356 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688496 4440 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:55:55.704356 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688499 4440 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:55:55.704356 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688501 4440 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:55:55.704356 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688504 4440 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:55:55.704356 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688506 4440 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:55:55.704924 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688509 4440 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:55:55.704924 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688511 4440 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:55:55.704924 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688514 4440 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:55:55.704924 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688517 4440 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:55:55.704924 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688520 4440 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:55:55.704924 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688522 4440 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:55:55.704924 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688525 4440 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:55:55.704924 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688528 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:55:55.704924 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688530 4440 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:55:55.704924 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688533 4440 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:55:55.704924 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688536 4440 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:55:55.704924 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688538 4440 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:55:55.704924 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688541 4440 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:55:55.704924 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.688566 4440 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:55:55.704924 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688603 4440 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:55:55.704924 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688608 4440 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:55:55.705362 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688611 4440 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:55:55.705362 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688614 4440 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:55:55.705362 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688617 4440 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:55:55.705362 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688620 4440 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:55:55.705362 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688623 4440 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:55:55.705362 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688625 4440 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:55:55.705362 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688628 4440 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:55:55.705362 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688631 4440 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:55:55.705362 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688634 4440 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:55:55.705362 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688636 4440 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:55:55.705362 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688640 4440 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:55:55.705362 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688643 4440 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:55:55.705362 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688646 4440 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:55:55.705362 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688648 4440 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:55:55.705362 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688651 4440 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:55:55.705362 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688653 4440 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:55:55.705362 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688656 4440 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:55:55.705362 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688659 4440 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:55:55.705362 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688661 4440 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:55:55.705362 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688664 4440 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:55:55.705985 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688667 4440 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:55:55.705985 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688669 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:55:55.705985 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688672 4440 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:55:55.705985 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688674 4440 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:55:55.705985 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688677 4440 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:55:55.705985 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688679 4440 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:55:55.705985 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688682 4440 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:55:55.705985 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688685 4440 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:55:55.705985 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688688 4440 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:55:55.705985 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688690 4440 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:55:55.705985 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688694 4440 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:55:55.705985 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688697 4440 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:55:55.705985 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688699 4440 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:55:55.705985 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688702 4440 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:55:55.705985 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688705 4440 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:55:55.705985 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688709 4440 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:55:55.705985 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688714 4440 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:55:55.705985 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688718 4440 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:55:55.705985 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688721 4440 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:55:55.706543 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688724 4440 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:55:55.706543 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688727 4440 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:55:55.706543 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688729 4440 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:55:55.706543 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688734 4440 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:55:55.706543 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688736 4440 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:55:55.706543 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688740 4440 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:55:55.706543 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688742 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:55:55.706543 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688745 4440 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:55:55.706543 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688747 4440 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:55:55.706543 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688750 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:55:55.706543 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688752 4440 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:55:55.706543 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688755 4440 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:55:55.706543 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688757 4440 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:55:55.706543 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688760 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:55:55.706543 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688763 4440 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:55:55.706543 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688766 4440 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:55:55.706543 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688768 4440 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:55:55.706543 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688771 4440 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:55:55.706543 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688773 4440 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:55:55.707065 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688776 4440 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:55:55.707065 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688779 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:55:55.707065 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688781 4440 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:55:55.707065 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688784 4440 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:55:55.707065 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688787 4440 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:55:55.707065 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688790 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:55:55.707065 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688792 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:55:55.707065 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688795 4440 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:55:55.707065 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688797 4440 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:55:55.707065 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688800 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:55:55.707065 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688803 4440 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:55:55.707065 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688805 4440 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:55:55.707065 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688808 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:55:55.707065 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688811 4440 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:55:55.707065 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688813 4440 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:55:55.707065 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688816 4440 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:55:55.707065 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688820 4440 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:55:55.707065 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688822 4440 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:55:55.707065 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688825 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:55:55.707065 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688827 4440 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:55:55.707631 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688830 4440 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:55:55.707631 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688832 4440 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:55:55.707631 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688835 4440 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:55:55.707631 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688838 4440 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:55:55.707631 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688840 4440 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:55:55.707631 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688843 4440 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:55:55.707631 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688845 4440 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:55:55.707631 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688848 4440 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:55:55.707631 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688851 4440 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:55:55.707631 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688854 4440 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:55:55.707631 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688856 4440 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:55:55.707631 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688859 4440 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:55:55.707631 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688861 4440 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:55:55.707631 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688865 4440 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:55:55.707631 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688867 4440 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:55:55.707631 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688870 4440 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:55:55.707631 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688873 4440 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:55:55.707631 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688875 4440 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:55:55.707631 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688878 4440 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:55:55.707631 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688881 4440 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:55:55.708126 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688883 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:55:55.708126 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688886 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:55:55.708126 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688888 4440 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:55:55.708126 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688895 4440 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:55:55.708126 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688898 4440 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:55:55.708126 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.688900 4440 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:55:55.708126 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.695245 4440 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 00:55:55.708126 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.695270 4440 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 00:55:55.708126 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.695299 4440 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 00:55:55.708126 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.695307 4440 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 00:55:55.708126 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695399 4440 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:55:55.708126 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695408 4440 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:55:55.708126 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695429 4440 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:55:55.708126 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695435 4440 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:55:55.708126 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695440 4440 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:55:55.708126 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695444 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:55:55.708126 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695449 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:55:55.708126 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695454 4440 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:55:55.708126 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695457 4440 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:55:55.708609 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695462 4440 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:55:55.708609 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695466 4440 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:55:55.708609 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695471 4440 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:55:55.708609 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695476 4440 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:55:55.708609 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695480 4440 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:55:55.708609 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695486 4440 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:55:55.708609 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695499 4440 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:55:55.708609 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695504 4440 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:55:55.708609 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695509 4440 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:55:55.708609 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695514 4440 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:55:55.708609 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695518 4440 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:55:55.708609 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695522 4440 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:55:55.708609 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695527 4440 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:55:55.708609 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695532 4440 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:55:55.708609 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695537 4440 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:55:55.708609 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695541 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:55:55.708609 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695545 4440 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:55:55.708609 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695550 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:55:55.708609 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695555 4440 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:55:55.708609 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695560 4440 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:55:55.709084 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695564 4440 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:55:55.709084 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695568 4440 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:55:55.709084 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695573 4440 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:55:55.709084 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695577 4440 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:55:55.709084 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695581 4440 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:55:55.709084 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695588 4440 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:55:55.709084 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695594 4440 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:55:55.709084 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695599 4440 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:55:55.709084 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695604 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:55:55.709084 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695608 4440 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:55:55.709084 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695613 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:55:55.709084 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695617 4440 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:55:55.709084 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695622 4440 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:55:55.709084 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695626 4440 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:55:55.709084 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695631 4440 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:55:55.709084 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695636 4440 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:55:55.709084 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695640 4440 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:55:55.709084 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695646 4440 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:55:55.709084 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695650 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:55:55.709567 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695655 4440 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:55:55.709567 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695659 4440 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:55:55.709567 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695663 4440 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:55:55.709567 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695667 4440 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:55:55.709567 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695673 4440 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:55:55.709567 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695678 4440 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:55:55.709567 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695683 4440 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:55:55.709567 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695687 4440 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:55:55.709567 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695691 4440 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:55:55.709567 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695696 4440 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:55:55.709567 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695700 4440 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:55:55.709567 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695705 4440 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:55:55.709567 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695709 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:55:55.709567 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695713 4440 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:55:55.709567 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695717 4440 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:55:55.709567 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695721 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:55:55.709567 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695725 4440 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:55:55.709567 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695730 4440 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:55:55.709567 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695734 4440 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:55:55.710161 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695739 4440 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:55:55.710161 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695743 4440 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:55:55.710161 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695747 4440 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:55:55.710161 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695752 4440 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:55:55.710161 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695757 4440 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:55:55.710161 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695761 4440 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:55:55.710161 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695766 4440 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:55:55.710161 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695771 4440 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:55:55.710161 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695775 4440 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:55:55.710161 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695780 4440 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:55:55.710161 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695784 4440 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:55:55.710161 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695788 4440 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:55:55.710161 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695794 4440 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:55:55.710161 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695798 4440 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:55:55.710161 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695803 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:55:55.710161 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695807 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:55:55.710161 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695812 4440 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:55:55.710161 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695817 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:55:55.710161 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695821 4440 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:55:55.710161 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695826 4440 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:55:55.710743 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695830 4440 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:55:55.710743 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695834 4440 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:55:55.710743 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695839 4440 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:55:55.710743 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695843 4440 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:55:55.710743 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695849 4440 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:55:55.710743 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695853 4440 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:55:55.710743 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695858 4440 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:55:55.710743 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695862 4440 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:55:55.710743 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695869 4440 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:55:55.710743 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695877 4440 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:55:55.710743 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695882 4440 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:55:55.710743 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695887 4440 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:55:55.710743 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695891 4440 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:55:55.710743 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695896 4440 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:55:55.710743 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695900 4440 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:55:55.710743 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695904 4440 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:55:55.710743 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695909 4440 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:55:55.710743 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695913 4440 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:55:55.710743 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.695917 4440 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:55:55.711206 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.695963 4440 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:55:55.711206 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696231 4440 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:55:55.711206 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696242 4440 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:55:55.711206 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696248 4440 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:55:55.711206 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696255 4440 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:55:55.711206 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696263 4440 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:55:55.711206 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696270 4440 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:55:55.711206 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696276 4440 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:55:55.711206 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696281 4440 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:55:55.711206 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696285 4440 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:55:55.711206 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696290 4440 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:55:55.711206 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696295 4440 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:55:55.711206 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696299 4440 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:55:55.711206 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696303 4440 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:55:55.711206 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696308 4440 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:55:55.711206 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696313 4440 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:55:55.711635 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696317 4440 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:55:55.711635 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696321 4440 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:55:55.711635 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696325 4440 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:55:55.711635 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696329 4440 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:55:55.711635 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696334 4440 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:55:55.711635 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696338 4440 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:55:55.711635 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696342 4440 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:55:55.711635 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696346 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:55:55.711635 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696351 4440 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:55:55.711635 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696355 4440 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:55:55.711635 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696359 4440 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:55:55.711635 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696364 4440 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:55:55.711635 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696368 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:55:55.711635 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696372 4440 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:55:55.711635 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696377 4440 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:55:55.711635 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696381 4440 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:55:55.711635 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696385 4440 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:55:55.711635 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696389 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:55:55.711635 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696393 4440 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:55:55.712088 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696398 4440 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:55:55.712088 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696403 4440 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:55:55.712088 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696407 4440 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:55:55.712088 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696412 4440 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:55:55.712088 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696440 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:55:55.712088 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696445 4440 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:55:55.712088 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696449 4440 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:55:55.712088 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696453 4440 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:55:55.712088 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696458 4440 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:55:55.712088 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696462 4440 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:55:55.712088 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696467 4440 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:55:55.712088 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696471 4440 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:55:55.712088 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696475 4440 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:55:55.712088 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696480 4440 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:55:55.712088 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696484 4440 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:55:55.712088 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696488 4440 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:55:55.712088 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696493 4440 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:55:55.712088 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696497 4440 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:55:55.712088 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696501 4440 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:55:55.712088 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696506 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:55:55.712597 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696510 4440 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:55:55.712597 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696514 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:55:55.712597 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696518 4440 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:55:55.712597 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696522 4440 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:55:55.712597 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696527 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:55:55.712597 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696531 4440 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:55:55.712597 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696535 4440 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:55:55.712597 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696539 4440 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:55:55.712597 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696543 4440 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:55:55.712597 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696548 4440 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:55:55.712597 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696552 4440 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:55:55.712597 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696557 4440 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:55:55.712597 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696562 4440 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:55:55.712597 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696569 4440 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:55:55.712597 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696574 4440 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:55:55.712597 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696580 4440 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:55:55.712597 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696585 4440 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:55:55.712597 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696590 4440 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:55:55.712597 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696594 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:55:55.713047 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696599 4440 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:55:55.713047 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696603 4440 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:55:55.713047 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696608 4440 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:55:55.713047 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696612 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:55:55.713047 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696616 4440 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:55:55.713047 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696620 4440 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:55:55.713047 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696625 4440 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:55:55.713047 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696629 4440 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:55:55.713047 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696633 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:55:55.713047 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696638 4440 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:55:55.713047 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696642 4440 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:55:55.713047 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696646 4440 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:55:55.713047 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696650 4440 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:55:55.713047 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696655 4440 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:55:55.713047 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696659 4440 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:55:55.713047 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696664 4440 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:55:55.713047 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696668 4440 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:55:55.713047 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696673 4440 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:55:55.713047 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696677 4440 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:55:55.713047 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696682 4440 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:55:55.713600 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696686 4440 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:55:55.713600 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696690 4440 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:55:55.713600 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696694 4440 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:55:55.713600 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696699 4440 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:55:55.713600 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696703 4440 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:55:55.713600 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696708 4440 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:55:55.713600 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696712 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:55:55.713600 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696717 4440 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:55:55.713600 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696722 4440 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:55:55.713600 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696726 4440 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:55:55.713600 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696731 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:55:55.713600 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696736 4440 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:55:55.713600 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696740 4440 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:55:55.713600 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.696785 4440 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:55:55.713600 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696839 4440 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:55:55.713600 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696847 4440 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:55:55.714210 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696852 4440 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:55:55.714210 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696857 4440 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:55:55.714210 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696861 4440 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:55:55.714210 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696866 4440 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:55:55.714210 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696870 4440 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:55:55.714210 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696875 4440 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:55:55.714210 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696879 4440 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:55:55.714210 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696883 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:55:55.714210 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696888 4440 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:55:55.714210 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696892 4440 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:55:55.714210 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696897 4440 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:55:55.714210 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696901 4440 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:55:55.714210 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696905 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:55:55.714210 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696909 4440 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:55:55.714210 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696913 4440 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:55:55.714210 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696918 4440 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:55:55.714210 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696922 4440 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:55:55.714210 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696927 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:55:55.714210 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696931 4440 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:55:55.714955 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696935 4440 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:55:55.714955 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696941 4440 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:55:55.714955 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696945 4440 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:55:55.714955 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696949 4440 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:55:55.714955 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696954 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:55:55.714955 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696958 4440 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:55:55.714955 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696962 4440 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:55:55.714955 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696967 4440 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:55:55.714955 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696971 4440 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:55:55.714955 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696976 4440 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:55:55.714955 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696980 4440 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:55:55.714955 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696985 4440 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:55:55.714955 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696989 4440 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:55:55.714955 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696993 4440 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:55:55.714955 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.696997 4440 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:55:55.714955 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697002 4440 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:55:55.714955 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697006 4440 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:55:55.714955 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697010 4440 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:55:55.714955 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697015 4440 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:55:55.714955 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697019 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:55:55.715491 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697023 4440 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:55:55.715491 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697027 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:55:55.715491 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697032 4440 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:55:55.715491 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697037 4440 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:55:55.715491 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697041 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:55:55.715491 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697046 4440 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:55:55.715491 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697050 4440 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:55:55.715491 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697054 4440 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:55:55.715491 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697059 4440 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:55:55.715491 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697063 4440 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:55:55.715491 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697067 4440 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:55:55.715491 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697071 4440 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:55:55.715491 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697076 4440 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:55:55.715491 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697083 4440 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:55:55.715491 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697089 4440 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:55:55.715491 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697094 4440 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:55:55.715491 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697098 4440 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:55:55.715491 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697104 4440 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:55:55.715491 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697108 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:55:55.716000 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697112 4440 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:55:55.716000 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697116 4440 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:55:55.716000 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697121 4440 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:55:55.716000 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697126 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:55:55.716000 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697131 4440 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:55:55.716000 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697135 4440 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:55:55.716000 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697140 4440 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:55:55.716000 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697144 4440 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:55:55.716000 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697148 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:55:55.716000 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697153 4440 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:55:55.716000 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697157 4440 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:55:55.716000 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697162 4440 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:55:55.716000 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697166 4440 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:55:55.716000 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697170 4440 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:55:55.716000 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697174 4440 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:55:55.716000 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697179 4440 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:55:55.716000 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697183 4440 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:55:55.716000 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697187 4440 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:55:55.716000 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697192 4440 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:55:55.716000 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697196 4440 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:55:55.716500 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697200 4440 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:55:55.716500 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697204 4440 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:55:55.716500 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697209 4440 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:55:55.716500 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697213 4440 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:55:55.716500 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697218 4440 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:55:55.716500 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697222 4440 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:55:55.716500 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697227 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:55:55.716500 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697231 4440 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:55:55.716500 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697236 4440 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:55:55.716500 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697240 4440 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:55:55.716500 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697244 4440 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:55:55.716500 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697249 4440 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:55:55.716500 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697253 4440 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:55:55.716500 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697258 4440 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:55:55.716500 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697262 4440 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:55:55.716500 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697267 4440 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:55:55.716500 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697274 4440 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:55:55.716500 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697281 4440 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:55:55.716500 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697286 4440 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:55:55.716500 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697293 4440 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:55:55.716982 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697298 4440 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:55:55.716982 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697302 4440 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:55:55.716982 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697307 4440 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:55:55.716982 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697311 4440 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:55:55.716982 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697315 4440 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:55:55.716982 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:55:55.697319 4440 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:55:55.716982 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.697406 4440 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 00:55:55.716982 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.699054 4440 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 00:55:55.716982 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.699141 4440 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 00:55:55.716982 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.699441 4440 server.go:1014] "Starting client certificate rotation" Oct 31 00:55:55.716982 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.699530 4440 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 00:55:55.716982 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.699581 4440 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 00:50:15 +0000 UTC" deadline="2028-05-09 06:29:17.983575539 +0000 UTC" Oct 31 00:55:55.716982 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.699615 4440 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13349h33m22.283961948s" Oct 31 00:55:55.716982 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.700209 4440 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:55:55.716982 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.702006 4440 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:55:55.716982 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.703676 4440 log.go:25] "Validated CRI v1 runtime API" Oct 31 00:55:55.717397 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.705392 4440 log.go:25] "Validated CRI v1 image API" Oct 31 00:55:55.717397 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.706241 4440 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 00:55:55.717397 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.707951 4440 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d8d4d7a8-786a-4c78-b06d-075d767b5b21:/dev/nvme0n1p3 f7d2d5b4-2150-4078-ac14-7df7b7fc6241:/dev/nvme0n1p4] Oct 31 00:55:55.717526 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.707975 4440 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1a780d8ff00f4f054342bb3d9a6fa482cc171917eacbf946303c50dc465eebfe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1a780d8ff00f4f054342bb3d9a6fa482cc171917eacbf946303c50dc465eebfe/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4581a2b0496fd321747947f34a5456bc2a49e484303cb17f074217897d5eb5f2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4581a2b0496fd321747947f34a5456bc2a49e484303cb17f074217897d5eb5f2/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/461b4f157fce847de1f83cb10e8a77c9a2c95b98de007c48546443716ebefb36/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/461b4f157fce847de1f83cb10e8a77c9a2c95b98de007c48546443716ebefb36/userdata/shm major:0 minor:212 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5f23efb80a3b70539d7eddbd4ef57b59f0b90f62c7f55e0509ac94d636ae5698/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5f23efb80a3b70539d7eddbd4ef57b59f0b90f62c7f55e0509ac94d636ae5698/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6bf194b52ad24d2798d0b7cd0a2cb4b697c1e7a6a27d53f708f38fadedcdfe34/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6bf194b52ad24d2798d0b7cd0a2cb4b697c1e7a6a27d53f708f38fadedcdfe34/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8bff4825a84b63e6edcadd67b0db93a359a938cb9e14f8c638eee7815a23cd8c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8bff4825a84b63e6edcadd67b0db93a359a938cb9e14f8c638eee7815a23cd8c/userdata/shm major:0 minor:69 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a83111a61a2f22ec5001f9fa3036f6a7bd673feb30fc55e65ea40f609e2a974e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a83111a61a2f22ec5001f9fa3036f6a7bd673feb30fc55e65ea40f609e2a974e/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c29ec7474f3f41e866a1a4774fad1dc26dfb897dffc49268023274db226cdeb7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c29ec7474f3f41e866a1a4774fad1dc26dfb897dffc49268023274db226cdeb7/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cd31ba38051bd98153d6964a968a80f77392c88ca1a73ad450f84bf749ba8b49/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cd31ba38051bd98153d6964a968a80f77392c88ca1a73ad450f84bf749ba8b49/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d80b2fdc62f60d2d25cc7c426407eec80ec1fedfb57f6e806624299d38dc9a14/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d80b2fdc62f60d2d25cc7c426407eec80ec1fedfb57f6e806624299d38dc9a14/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e3d7e294ddd6e2bc291ade2bc81bae50733485bf3303acfd8fe2d4c137845eaa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e3d7e294ddd6e2bc291ade2bc81bae50733485bf3303acfd8fe2d4c137845eaa/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e819bed747b2de3500ac2a00a9d954a62304db15d628d5fbca0e83e9eafe8ad5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e819bed747b2de3500ac2a00a9d954a62304db15d628d5fbca0e83e9eafe8ad5/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ebd6b5435799a1093f7e6e89ffbc0ba1b578811f81bd0a71c16f532a75fd826f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ebd6b5435799a1093f7e6e89ffbc0ba1b578811f81bd0a71c16f532a75fd826f/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f26c472e51c7f868476b174a94eb7f7f44e59a358941d0a9d6133a1f25c73842/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f26c472e51c7f868476b174a94eb7f7f44e59a358941d0a9d6133a1f25c73842/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f3361e09423c13c5d72aa9f81c5fdd72aa2b6dea2802635f190aa391de176b5c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f3361e09423c13c5d72aa9f81c5fdd72aa2b6dea2802635f190aa391de176b5c/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a37f05b-0755-4ca7-8d9b-12030234bec1/volumes/kubernetes.io~projected/kube-api-access-npk86:{mountpoint:/var/lib/kubelet/pods/0a37f05b-0755-4ca7-8d9b-12030234bec1/volumes/kubernetes.io~projected/kube-api-access-npk86 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f/volumes/kubernetes.io~projected/kube-api-access-7nchj:{mountpoint:/var/lib/kubelet/pods/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f/volumes/kubernetes.io~projected/kube-api-access-7nchj major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12002bf6-fce5-4d20-a62c-6782522acd47/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/12002bf6-fce5-4d20-a62c-6782522acd47/volumes/kubernetes.io~secret/agent-certs major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d/volumes/kubernetes.io~projected/kube-api-access-fdbsq:{mountpoint:/var/lib/kubelet/pods/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d/volumes/kubernetes.io~projected/kube-api-access-fdbsq major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3c9845ea-a9d6-4d5c-93ea-fb096666a2b9/volumes/kubernetes.io~projected/kube-api-access-kwrz5:{mountpoint:/var/lib/kubelet/pods/3c9845ea-a9d6-4d5c-93ea-fb096666a2b9/volumes/kubernetes.io~projected/kube-api-access-kwrz5 major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49f7c8f4-b979-436d-b035-20f9fcae140a/volumes/kubernetes.io~projected/kube-api-access-vzddr:{mountpoint:/var/lib/kubelet/pods/49f7c8f4-b979-436d-b035-20f9fcae140a/volumes/kubernetes.io~projected/kube-api-access-vzddr major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49f8a438-6b29-4160-98e1-3f26e27fe0af/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/49f8a438-6b29-4160-98e1-3f26e27fe0af/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b8ba0db-d8b9-4a40-a144-1400634cd99d/volumes/kubernetes.io~projected/kube-api-access-t9v2m:{mountpoint:/var/lib/kubelet/pods/4b8ba0db-d8b9-4a40-a144-1400634cd99d/volumes/kubernetes.io~projected/kube-api-access-t9v2m major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88c1a03a-cb94-4226-9d53-cd923895569e/volumes/kubernetes.io~projected/kube-api-access-986bh:{mountpoint:/var/lib/kubelet/pods/88c1a03a-cb94-4226-9d53-cd923895569e/volumes/kubernetes.io~projected/kube-api-access-986bh major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a11b541-6b88-439e-bcbe-5901cb4c20b0/volumes/kubernetes.io~projected/kube-api-access-hnc9j:{mountpoint:/var/lib/kubelet/pods/9a11b541-6b88-439e-bcbe-5901cb4c20b0/volumes/kubernetes.io~projected/kube-api-access-hnc9j major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a11b541-6b88-439e-bcbe-5901cb4c20b0/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/9a11b541-6b88-439e-bcbe-5901cb4c20b0/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6ddfc5f-887b-4821-bac9-54cb59c50e32/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/a6ddfc5f-887b-4821-bac9-54cb59c50e32/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6ddfc5f-887b-4821-bac9-54cb59c50e32/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/a6ddfc5f-887b-4821-bac9-54cb59c50e32/volumes/kubernetes.io~empty-dir/tmp major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6ddfc5f-887b-4821-bac9-54cb59c50e32/volumes/kubernetes.io~projected/kube-api-access-fnvxk:{mountpoint:/var/lib/kubelet/pods/a6ddfc5f-887b-4821-bac9-54cb59c50e32/volumes/kubernetes.io~projected/kube-api-access-fnvxk major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac0f8a44-5a66-40fe-a08c-080314e7ea60/volumes/kubernetes.io~projected/kube-api-access-hfhcf:{mountpoint:/var/lib/kubelet/pods/ac0f8a44-5a66-40fe-a08c-080314e7ea60/volumes/kubernetes.io~projected/kube-api-access-hfhcf major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c3939885-f5d4-4bd2-a69f-435c2a8fa048/volumes/kubernetes.io~projected/kube-api-access-4j6vc:{mountpoint:/var/lib/kubelet/pods/c3939885-f5d4-4bd2-a69f-435c2a8fa048/volumes/kubernetes.io~projected/kube-api-access-4j6vc major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd654994-98a0-4d39-aa38-270b4a0d4341/volumes/kubernetes.io~projected/kube-api-access-m4qw7:{mountpoint:/var/lib/kubelet/pods/cd654994-98a0-4d39-aa38-270b4a0d4341/volumes/kubernetes.io~projected/kube-api-access-m4qw7 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d610021a-d4d9-473d-8889-cb5afcb5b8a0/volumes/kubernetes.io~projected/kube-api-access-xs6kv:{mountpoint:/var/lib/kubelet/pods/d610021a-d4d9-473d-8889-cb5afcb5b8a0/volumes/kubernetes.io~projected/kube-api-access-xs6kv major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee8bc72c-f181-452d-ab71-b13a51672bcf/volumes/kubernetes.io~projected/kube-api-access-klvmv:{mountpoint:/var/lib/kubelet/pods/ee8bc72c-f181-452d-ab71-b13a51672bcf/volumes/kubernetes.io~projected/kube-api-access-klvmv major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee8bc72c-f181-452d-ab71-b13a51672bcf/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/ee8bc72c-f181-452d-ab71-b13a51672bcf/volumes/kubernetes.io~secret/signing-key major:0 minor:223 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/559dad106ec67acfc302d45bc81dd3aa09669ea2bdc7ccf8b9c481d05a9cd991/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/fbb1e99adf041e60e639d2446544032e840ab726c12e3b73ab0acd50878db568/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/3fdf44817fcc5db07322654b1894b7b20f8328c7b3b780fcb317eded549499d9/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/8540b2b48a71ddeb2d99ea8738d914ed079811c4af229224ca9cf12731929fc4/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/05d95f903b79d7d560c4c3787793b0404445b9fdd706a3d71e58b47fb2a44ef4/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/7c7f46bf43b37c8862a9c14fbdac59b253fd9ca78877f96b9626cc3e5a803534/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/d917b0931281e55f0e2680a111f86f39aa122686db940429c39bd4917e046f8f/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/eabb69fca4ce3d96c1cc4854dea3166eb58b13da236276ff70363a82bedcc537/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/3c3271a892c69ac14bd27d335199bbb05713a86da2f7c2c530f0cd24ad33e809/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/6f919e79a40987bf5eb3dcc59f095bfa141c3cfd9a8b195743eab2175843f9cf/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/2ea03e7aa28ec8d36bcc2ca12120167623146fd26d97a7cadc780fcdbbf19b9f/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/8c9a0d9c38b0910553d11139e47c6fdec47219922d9c70df0d6f71bcc547950c/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/d25a172e6d4bdef0d553fe9112559fd382e15b03faafe45617357427636b673f/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-155:{mountpoint:/var/lib/containers/storage/overlay/658591b945c9e0c31c7ce7cfa4c85dac3c1d32908ca7d09af558beef00753208/merged major:0 minor:155 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/649047b590489f9bf34b494a53c66134ccf281fbfa718727b4fafe96d5027d4e/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/757ba6e31dfd3fc9a5cdfb83bd3d508c2ef26ade3f914b9840e2ccf0b6368dea/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/92e77412d39cd78878f1c94ee7b924ce9374c0d42e59a562ec09e53a0bafec25/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/bf906509a32d90d05a01a8681d65ec6027b74e6aeac4c7dd2a9a0ed57e670d25/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/60dac6af3acf1687a3c9aa8e65597aba54698de758d22e6c637018162a583d12/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/b720aebefa7c033403bafe6f934a2d540ce6ce11849d92b340f6ba4f6c75f689/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/9b471f12bed2cfb12cccb9590234a4449af77f2200beda7e632eb075c84045f0/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/153ef237935c4ed40d46842b8e5201f2bd028a1f0855aab22b00fd2823be51b6/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-210:{mountpoint:/var/lib/containers/storage/overlay/c98d6266b3cd40e139b76c543937165af29e794c6a354c6ad8fbb8367f4f640d/merged major:0 minor:210 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/2493b9fc9e657f7a8a1e4df431405d2121e319f9f41e19c5f8effdff062bb905/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/77b7acb7e8369f5433c145ff050f2a62a9418ffa0dcbf34c0ab00ead9385aadd/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-221:{mountpoint:/var/lib/containers/storage/overlay/480b7e00f95c79c5e173eb45ee264375f56d508a10b93bff64057f615ccee2c9/merged major:0 minor:221 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/7a915b884c3e314ee224693dabbcae49ee687d3d15aa5e40be8b970084f65a27/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/aad854b6aa470479d1d53e52d3d5a8748e3cca2f39a428ddec9b37862a91f8d2/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/ebaf877128163326f9447652b879cdb5a30197fa59a8d04d3f7cd367524f9ad6/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/3c9acf9bcb62d495bd988d41aebc3ef8e4508399bbd1e4b9bb6db91ec29e8542/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/4d6e6046eb2a46968c33123b9117529b37d5d5153b33164328f214ca3971c90f/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/cc30aff3aa1b349701e901901d758201c75d3a7162a3c5b7e04f9d9009d31043/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-71:{mountpoint:/var/lib/containers/storage/overlay/3a976175e5911b171439c73ea494afea8c8e51cf9e4b6e60dd33b97461ff6ffe/merged major:0 minor:71 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/bf105662bc7422b5e2780daaa7365739be6e76f42e3e0173f5396454e90b6047/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/3b18a62ca96770c90e7a565e8a05e9d33bb6be40260c792b26aa4469d14aa839/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/5da4d69eeb06fc07f0631fb8242a24b221db49365f5f1315d433156c86d6ec3f/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/9fe8caf7110255f3172f3eabb214c77fcdbc32f46a7312957c7c8234cb133dd3/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/1a9781ef9192079151b91d0a2869e78b879c35332c11b45082608bbeeabb8c5a/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/3652b477fde0d84c975bc90962a44b71b3129c38afa6a61d79bf5a0c70a01702/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/0c3c4c4ad210f2f9bdb8797eece4b4859b99b812ac1a2684a37504c5bcbcf773/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 31 00:55:55.717673 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.715264 4440 manager.go:217] Machine: {Timestamp:2026-10-31 00:55:55.714723133 +0000 UTC m=+0.073445442 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099979 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec223bd617052241626d969ccd19a66a SystemUUID:ec223bd6-1705-2241-626d-969ccd19a66a BootID:3df2f492-315d-4014-9f17-4520f8fd14f3 Filesystems:[{Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9a11b541-6b88-439e-bcbe-5901cb4c20b0/volumes/kubernetes.io~projected/kube-api-access-hnc9j DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-221 DeviceMajor:0 DeviceMinor:221 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d/volumes/kubernetes.io~projected/kube-api-access-fdbsq DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6bf194b52ad24d2798d0b7cd0a2cb4b697c1e7a6a27d53f708f38fadedcdfe34/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a6ddfc5f-887b-4821-bac9-54cb59c50e32/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-155 DeviceMajor:0 DeviceMinor:155 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4b8ba0db-d8b9-4a40-a144-1400634cd99d/volumes/kubernetes.io~projected/kube-api-access-t9v2m DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cd654994-98a0-4d39-aa38-270b4a0d4341/volumes/kubernetes.io~projected/kube-api-access-m4qw7 DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5f23efb80a3b70539d7eddbd4ef57b59f0b90f62c7f55e0509ac94d636ae5698/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ee8bc72c-f181-452d-ab71-b13a51672bcf/volumes/kubernetes.io~projected/kube-api-access-klvmv DeviceMajor:0 DeviceMinor:228 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d610021a-d4d9-473d-8889-cb5afcb5b8a0/volumes/kubernetes.io~projected/kube-api-access-xs6kv DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f/volumes/kubernetes.io~projected/kube-api-access-7nchj DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ebd6b5435799a1093f7e6e89ffbc0ba1b578811f81bd0a71c16f532a75fd826f/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f3361e09423c13c5d72aa9f81c5fdd72aa2b6dea2802635f190aa391de176b5c/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e819bed747b2de3500ac2a00a9d954a62304db15d628d5fbca0e83e9eafe8ad5/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/12002bf6-fce5-4d20-a62c-6782522acd47/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a83111a61a2f22ec5001f9fa3036f6a7bd673feb30fc55e65ea40f609e2a974e/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/461b4f157fce847de1f83cb10e8a77c9a2c95b98de007c48546443716ebefb36/userdata/shm DeviceMajor:0 DeviceMinor:212 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cd31ba38051bd98153d6964a968a80f77392c88ca1a73ad450f84bf749ba8b49/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1a780d8ff00f4f054342bb3d9a6fa482cc171917eacbf946303c50dc465eebfe/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-210 DeviceMajor:0 DeviceMinor:210 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c3939885-f5d4-4bd2-a69f-435c2a8fa048/volumes/kubernetes.io~projected/kube-api-access-4j6vc DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4581a2b0496fd321747947f34a5456bc2a49e484303cb17f074217897d5eb5f2/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f26c472e51c7f868476b174a94eb7f7f44e59a358941d0a9d6133a1f25c73842/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/49f8a438-6b29-4160-98e1-3f26e27fe0af/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3c9845ea-a9d6-4d5c-93ea-fb096666a2b9/volumes/kubernetes.io~projected/kube-api-access-kwrz5 DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a6ddfc5f-887b-4821-bac9-54cb59c50e32/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-71 DeviceMajor:0 DeviceMinor:71 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9a11b541-6b88-439e-bcbe-5901cb4c20b0/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c29ec7474f3f41e866a1a4774fad1dc26dfb897dffc49268023274db226cdeb7/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ee8bc72c-f181-452d-ab71-b13a51672bcf/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d80b2fdc62f60d2d25cc7c426407eec80ec1fedfb57f6e806624299d38dc9a14/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ac0f8a44-5a66-40fe-a08c-080314e7ea60/volumes/kubernetes.io~projected/kube-api-access-hfhcf DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e3d7e294ddd6e2bc291ade2bc81bae50733485bf3303acfd8fe2d4c137845eaa/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/88c1a03a-cb94-4226-9d53-cd923895569e/volumes/kubernetes.io~projected/kube-api-access-986bh DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a6ddfc5f-887b-4821-bac9-54cb59c50e32/volumes/kubernetes.io~projected/kube-api-access-fnvxk DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8bff4825a84b63e6edcadd67b0db93a359a938cb9e14f8c638eee7815a23cd8c/userdata/shm DeviceMajor:0 DeviceMinor:69 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/49f7c8f4-b979-436d-b035-20f9fcae140a/volumes/kubernetes.io~projected/kube-api-access-vzddr DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0a37f05b-0755-4ca7-8d9b-12030234bec1/volumes/kubernetes.io~projected/kube-api-access-npk86 DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1a780d8ff00f4f0 MacAddress:96:8a:28:95:69:5a Speed:10000 Mtu:8901} {Name:461b4f157fce847 MacAddress:4e:1b:4b:79:95:4c Speed:10000 Mtu:8901} {Name:a83111a61a2f22e MacAddress:12:b5:e7:a7:a6:b9 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:da:8f:44:23 Speed:0 Mtu:9001} {Name:br-int MacAddress:ca:99:87:aa:b5:57 Speed:0 Mtu:8901} {Name:c29ec7474f3f41e MacAddress:82:04:5d:5c:14:34 Speed:10000 Mtu:8901} {Name:e3d7e294ddd6e2b MacAddress:f6:6b:ee:e4:04:b8 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:da:8f:44:23 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:b2:59:e6:fc:fe:c1 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:aa:13:c6:01:01:4d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 00:55:55.717673 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.716135 4440 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 00:55:55.717673 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.716213 4440 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 00:55:55.717673 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.716736 4440 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 00:55:55.717673 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.716764 4440 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-39.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 00:55:55.717673 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.716903 4440 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 00:55:55.717673 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.716909 4440 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 00:55:55.717673 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.716920 4440 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:55:55.717673 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.716940 4440 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:55:55.717673 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.717047 4440 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 00:55:55.717673 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.717218 4440 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 00:55:55.717673 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.717329 4440 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 00:55:55.717673 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.717366 4440 kubelet.go:493] "Attempting to sync node with API server" Oct 31 00:55:55.717673 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.717375 4440 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 00:55:55.717673 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.717385 4440 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 00:55:55.717673 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.717395 4440 kubelet.go:405] "Adding apiserver pod source" Oct 31 00:55:55.717673 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.717402 4440 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 00:55:55.718348 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.718069 4440 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 00:55:55.718348 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.718078 4440 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:55:55.718722 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.718698 4440 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 00:55:55.727707 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.727679 4440 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 00:55:55.727707 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.727696 4440 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:55:55.727887 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.727821 4440 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 00:55:55.728109 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.728093 4440 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 00:55:55.728178 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.728133 4440 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 00:55:55.728273 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.728262 4440 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 00:55:55.728325 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.728281 4440 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 00:55:55.728325 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.728292 4440 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 00:55:55.728325 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.728301 4440 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 00:55:55.728325 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.728310 4440 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 00:55:55.728325 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.728320 4440 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 00:55:55.728605 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.728330 4440 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 00:55:55.728605 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.728339 4440 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 00:55:55.728605 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.728349 4440 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 00:55:55.728605 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.728359 4440 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 00:55:55.728605 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.728380 4440 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 00:55:55.728605 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.728394 4440 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 00:55:55.728605 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.728495 4440 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 00:55:55.728605 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.728506 4440 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 00:55:55.732110 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.732090 4440 server.go:1263] "Started kubelet" Oct 31 00:55:55.732209 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.732124 4440 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:55:55.732372 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.732308 4440 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 00:55:55.732457 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.732362 4440 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 00:55:55.732457 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.732405 4440 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 00:55:55.732632 ip-10-0-130-39 systemd[1]: Started Kubernetes Kubelet. Oct 31 00:55:55.732791 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.732772 4440 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 00:55:55.734302 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.734270 4440 server.go:317] "Adding debug handlers to kubelet server" Oct 31 00:55:55.735361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.735336 4440 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:55:55.738548 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.736301 4440 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 00:55:55.738548 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.736659 4440 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 00:55:55.738548 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.737230 4440 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 00:55:55.738548 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.737244 4440 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 00:55:55.738548 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.737553 4440 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 00:55:55.738548 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.737854 4440 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:50:17 +0000 UTC" deadline="2028-06-24 11:42:52.615522165 +0000 UTC" Oct 31 00:55:55.738548 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.737897 4440 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14458h46m56.877629289s" Oct 31 00:55:55.739191 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.739170 4440 factory.go:55] Registering systemd factory Oct 31 00:55:55.739280 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.739194 4440 factory.go:223] Registration of the systemd container factory successfully Oct 31 00:55:55.741054 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741036 4440 factory.go:153] Registering CRI-O factory Oct 31 00:55:55.741054 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741057 4440 factory.go:223] Registration of the crio container factory successfully Oct 31 00:55:55.741357 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741052 4440 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6ddfc5f-887b-4821-bac9-54cb59c50e32" volumeName="kubernetes.io/projected/a6ddfc5f-887b-4821-bac9-54cb59c50e32-kube-api-access-fnvxk" seLinuxMountContext="" Oct 31 00:55:55.741357 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741093 4440 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49f8a438-6b29-4160-98e1-3f26e27fe0af" volumeName="kubernetes.io/secret/49f8a438-6b29-4160-98e1-3f26e27fe0af-original-pull-secret" seLinuxMountContext="" Oct 31 00:55:55.741357 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741109 4440 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6ddfc5f-887b-4821-bac9-54cb59c50e32" volumeName="kubernetes.io/empty-dir/a6ddfc5f-887b-4821-bac9-54cb59c50e32-tmp" seLinuxMountContext="" Oct 31 00:55:55.741357 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741121 4440 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 00:55:55.741357 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741122 4440 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d610021a-d4d9-473d-8889-cb5afcb5b8a0" volumeName="kubernetes.io/projected/d610021a-d4d9-473d-8889-cb5afcb5b8a0-kube-api-access-xs6kv" seLinuxMountContext="" Oct 31 00:55:55.741357 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741137 4440 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d" volumeName="kubernetes.io/configmap/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-ovnkube-script-lib" seLinuxMountContext="" Oct 31 00:55:55.741357 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741073 4440 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:55:55.741357 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741150 4440 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49f7c8f4-b979-436d-b035-20f9fcae140a" volumeName="kubernetes.io/projected/49f7c8f4-b979-436d-b035-20f9fcae140a-kube-api-access-vzddr" seLinuxMountContext="" Oct 31 00:55:55.741357 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741200 4440 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b8ba0db-d8b9-4a40-a144-1400634cd99d" volumeName="kubernetes.io/projected/4b8ba0db-d8b9-4a40-a144-1400634cd99d-kube-api-access-t9v2m" seLinuxMountContext="" Oct 31 00:55:55.741357 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741222 4440 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d610021a-d4d9-473d-8889-cb5afcb5b8a0" volumeName="kubernetes.io/configmap/d610021a-d4d9-473d-8889-cb5afcb5b8a0-cni-binary-copy" seLinuxMountContext="" Oct 31 00:55:55.741357 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741238 4440 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee8bc72c-f181-452d-ab71-b13a51672bcf" volumeName="kubernetes.io/projected/ee8bc72c-f181-452d-ab71-b13a51672bcf-kube-api-access-klvmv" seLinuxMountContext="" Oct 31 00:55:55.741357 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741252 4440 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c9845ea-a9d6-4d5c-93ea-fb096666a2b9" volumeName="kubernetes.io/projected/3c9845ea-a9d6-4d5c-93ea-fb096666a2b9-kube-api-access-kwrz5" seLinuxMountContext="" Oct 31 00:55:55.741357 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741264 4440 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88c1a03a-cb94-4226-9d53-cd923895569e" volumeName="kubernetes.io/configmap/88c1a03a-cb94-4226-9d53-cd923895569e-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 00:55:55.741357 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741276 4440 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a37f05b-0755-4ca7-8d9b-12030234bec1" volumeName="kubernetes.io/configmap/0a37f05b-0755-4ca7-8d9b-12030234bec1-serviceca" seLinuxMountContext="" Oct 31 00:55:55.741357 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741289 4440 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fe8107c-9128-4c8a-9bde-e1cacd2ed19f" volumeName="kubernetes.io/projected/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f-kube-api-access-7nchj" seLinuxMountContext="" Oct 31 00:55:55.741357 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741300 4440 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d" volumeName="kubernetes.io/secret/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 00:55:55.741357 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741140 4440 factory.go:103] Registering Raw factory Oct 31 00:55:55.741357 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741316 4440 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49f7c8f4-b979-436d-b035-20f9fcae140a" volumeName="kubernetes.io/configmap/49f7c8f4-b979-436d-b035-20f9fcae140a-config-volume" seLinuxMountContext="" Oct 31 00:55:55.741357 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741353 4440 manager.go:1199] Started watching for new ooms in manager Oct 31 00:55:55.741357 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741353 4440 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b8ba0db-d8b9-4a40-a144-1400634cd99d" volumeName="kubernetes.io/configmap/4b8ba0db-d8b9-4a40-a144-1400634cd99d-iptables-alerter-script" seLinuxMountContext="" Oct 31 00:55:55.742313 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741387 4440 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88c1a03a-cb94-4226-9d53-cd923895569e" volumeName="kubernetes.io/configmap/88c1a03a-cb94-4226-9d53-cd923895569e-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 00:55:55.742313 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741402 4440 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88c1a03a-cb94-4226-9d53-cd923895569e" volumeName="kubernetes.io/projected/88c1a03a-cb94-4226-9d53-cd923895569e-kube-api-access-986bh" seLinuxMountContext="" Oct 31 00:55:55.742313 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741433 4440 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d" volumeName="kubernetes.io/configmap/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-ovnkube-config" seLinuxMountContext="" Oct 31 00:55:55.742313 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741445 4440 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49f7c8f4-b979-436d-b035-20f9fcae140a" volumeName="kubernetes.io/empty-dir/49f7c8f4-b979-436d-b035-20f9fcae140a-tmp-dir" seLinuxMountContext="" Oct 31 00:55:55.742313 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741464 4440 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6ddfc5f-887b-4821-bac9-54cb59c50e32" volumeName="kubernetes.io/empty-dir/a6ddfc5f-887b-4821-bac9-54cb59c50e32-etc-tuned" seLinuxMountContext="" Oct 31 00:55:55.742313 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741476 4440 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee8bc72c-f181-452d-ab71-b13a51672bcf" volumeName="kubernetes.io/configmap/ee8bc72c-f181-452d-ab71-b13a51672bcf-signing-cabundle" seLinuxMountContext="" Oct 31 00:55:55.742313 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741490 4440 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88c1a03a-cb94-4226-9d53-cd923895569e" volumeName="kubernetes.io/configmap/88c1a03a-cb94-4226-9d53-cd923895569e-cni-binary-copy" seLinuxMountContext="" Oct 31 00:55:55.742313 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741515 4440 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a11b541-6b88-439e-bcbe-5901cb4c20b0" volumeName="kubernetes.io/projected/9a11b541-6b88-439e-bcbe-5901cb4c20b0-kube-api-access-hnc9j" seLinuxMountContext="" Oct 31 00:55:55.742313 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741523 4440 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac0f8a44-5a66-40fe-a08c-080314e7ea60" volumeName="kubernetes.io/projected/ac0f8a44-5a66-40fe-a08c-080314e7ea60-kube-api-access-hfhcf" seLinuxMountContext="" Oct 31 00:55:55.742313 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741553 4440 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd654994-98a0-4d39-aa38-270b4a0d4341" volumeName="kubernetes.io/projected/cd654994-98a0-4d39-aa38-270b4a0d4341-kube-api-access-m4qw7" seLinuxMountContext="" Oct 31 00:55:55.742313 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741573 4440 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d610021a-d4d9-473d-8889-cb5afcb5b8a0" volumeName="kubernetes.io/configmap/d610021a-d4d9-473d-8889-cb5afcb5b8a0-multus-daemon-config" seLinuxMountContext="" Oct 31 00:55:55.742313 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741582 4440 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee8bc72c-f181-452d-ab71-b13a51672bcf" volumeName="kubernetes.io/secret/ee8bc72c-f181-452d-ab71-b13a51672bcf-signing-key" seLinuxMountContext="" Oct 31 00:55:55.742313 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741595 4440 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a37f05b-0755-4ca7-8d9b-12030234bec1" volumeName="kubernetes.io/projected/0a37f05b-0755-4ca7-8d9b-12030234bec1-kube-api-access-npk86" seLinuxMountContext="" Oct 31 00:55:55.742313 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741605 4440 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12002bf6-fce5-4d20-a62c-6782522acd47" volumeName="kubernetes.io/configmap/12002bf6-fce5-4d20-a62c-6782522acd47-konnectivity-ca" seLinuxMountContext="" Oct 31 00:55:55.742313 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741612 4440 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d" volumeName="kubernetes.io/configmap/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-env-overrides" seLinuxMountContext="" Oct 31 00:55:55.742313 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741628 4440 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a11b541-6b88-439e-bcbe-5901cb4c20b0" volumeName="kubernetes.io/secret/9a11b541-6b88-439e-bcbe-5901cb4c20b0-cert" seLinuxMountContext="" Oct 31 00:55:55.742313 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741638 4440 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3939885-f5d4-4bd2-a69f-435c2a8fa048" volumeName="kubernetes.io/projected/c3939885-f5d4-4bd2-a69f-435c2a8fa048-kube-api-access-4j6vc" seLinuxMountContext="" Oct 31 00:55:55.742313 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741645 4440 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd654994-98a0-4d39-aa38-270b4a0d4341" volumeName="kubernetes.io/empty-dir/cd654994-98a0-4d39-aa38-270b4a0d4341-tmp-dir" seLinuxMountContext="" Oct 31 00:55:55.742313 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741652 4440 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12002bf6-fce5-4d20-a62c-6782522acd47" volumeName="kubernetes.io/secret/12002bf6-fce5-4d20-a62c-6782522acd47-agent-certs" seLinuxMountContext="" Oct 31 00:55:55.742313 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741659 4440 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d" volumeName="kubernetes.io/projected/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-kube-api-access-fdbsq" seLinuxMountContext="" Oct 31 00:55:55.742313 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741666 4440 reconstruct.go:97] "Volume reconstruction finished" Oct 31 00:55:55.742313 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741671 4440 reconciler.go:29] "Reconciler: start to sync state" Oct 31 00:55:55.742313 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.741689 4440 manager.go:319] Starting recovery of all containers Oct 31 00:55:55.744801 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.744784 4440 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 00:55:55.750612 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.750546 4440 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 00:55:55.763313 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.763284 4440 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 00:55:55.763313 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.763310 4440 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 00:55:55.763541 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.763330 4440 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 00:55:55.763541 ip-10-0-130-39 kubenswrapper[4440]: E1031 00:55:55.763377 4440 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 00:55:55.767173 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.767125 4440 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:55:55.784575 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.784535 4440 generic.go:356] "Generic (PLEG): container finished" podID="88c1a03a-cb94-4226-9d53-cd923895569e" containerID="04099c4cd5dfe5c4572cd656ebf34bb77c5da6522bdae6a84444b7ee45bfb9c4" exitCode=0 Oct 31 00:55:55.784575 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.784568 4440 generic.go:356] "Generic (PLEG): container finished" podID="88c1a03a-cb94-4226-9d53-cd923895569e" containerID="db051957365c06b73c8dda860f6f158e54f9862e6de96cc1a3b09cf93518ccc2" exitCode=0 Oct 31 00:55:55.784575 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.784574 4440 generic.go:356] "Generic (PLEG): container finished" podID="88c1a03a-cb94-4226-9d53-cd923895569e" containerID="cf91db1cbe3eeb915ca5b3ac11916ff33583863a26368d54e828df2b03e07a9f" exitCode=0 Oct 31 00:55:55.784575 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.784583 4440 generic.go:356] "Generic (PLEG): container finished" podID="88c1a03a-cb94-4226-9d53-cd923895569e" containerID="0090b900cfe0d52499a44a37f4ec26fd067884b6f9d958abc1bd0c4592521d4e" exitCode=0 Oct 31 00:55:55.784892 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.784590 4440 generic.go:356] "Generic (PLEG): container finished" podID="88c1a03a-cb94-4226-9d53-cd923895569e" containerID="803a696233b5fef7d4370215e02d38f4aed242cbcf59290b6e750c9f4fac8097" exitCode=0 Oct 31 00:55:55.784892 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.784600 4440 generic.go:356] "Generic (PLEG): container finished" podID="88c1a03a-cb94-4226-9d53-cd923895569e" containerID="e0aa8d81ae98b35c2c8df1c330db064a0fd8180b04556aefbdcfcc556bb9d5c4" exitCode=0 Oct 31 00:55:55.786494 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.786465 4440 generic.go:356] "Generic (PLEG): container finished" podID="ce6a902de7718819f66be8130076b8e6" containerID="4e433a1feb85eed73620569224ecdabfb4a5ce29eb2412c13506fd108fd8dbf2" exitCode=0 Oct 31 00:55:55.806976 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.806953 4440 manager.go:324] Recovery completed Oct 31 00:55:55.821717 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.821645 4440 cpu_manager.go:226] "Starting" policy="none" Oct 31 00:55:55.821717 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.821666 4440 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 00:55:55.821717 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.821685 4440 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 00:55:55.821880 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.821815 4440 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 00:55:55.821880 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.821826 4440 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 00:55:55.821880 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.821843 4440 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 00:55:55.821880 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.821848 4440 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 00:55:55.821880 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.821854 4440 policy_none.go:50] "Start" Oct 31 00:55:55.821880 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.821862 4440 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 00:55:55.821880 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.821870 4440 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 00:55:55.822071 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.821998 4440 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 00:55:55.822071 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.822005 4440 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 00:55:55.822071 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.822010 4440 policy_none.go:44] "Start" Oct 31 00:55:55.826537 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.826507 4440 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 00:55:55.829070 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.829054 4440 manager.go:341] "Starting Device Plugin manager" Oct 31 00:55:55.829189 ip-10-0-130-39 kubenswrapper[4440]: E1031 00:55:55.829086 4440 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 00:55:55.829189 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.829097 4440 server.go:87] "Starting device plugin registration server" Oct 31 00:55:55.829433 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.829408 4440 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 00:55:55.829508 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.829436 4440 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 00:55:55.829554 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.829533 4440 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 00:55:55.829657 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.829622 4440 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 00:55:55.829699 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.829666 4440 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 00:55:55.829699 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.829674 4440 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 00:55:55.829782 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.829716 4440 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T00:55:55.829655467Z","UUID":"a257821b-7bd6-4444-a4f4-bdb9778d0e84","Handler":null,"Name":"","Endpoint":""} Oct 31 00:55:55.831013 ip-10-0-130-39 kubenswrapper[4440]: E1031 00:55:55.830973 4440 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 00:55:55.831402 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.831385 4440 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 00:55:55.831519 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.831410 4440 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 00:55:55.863744 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.863679 4440 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-39.us-east-2.compute.internal"] Oct 31 00:55:55.864246 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.864202 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal" event={"ID":"ce6a902de7718819f66be8130076b8e6","Type":"ContainerStarted","Data":"e3611575ec54eb27b687d7eafd5116a93e852f66843821402196e9b29faab55f"} Oct 31 00:55:55.864246 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.864249 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal" event={"ID":"ce6a902de7718819f66be8130076b8e6","Type":"ContainerDied","Data":"4e433a1feb85eed73620569224ecdabfb4a5ce29eb2412c13506fd108fd8dbf2"} Oct 31 00:55:55.864458 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.864270 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal" event={"ID":"ce6a902de7718819f66be8130076b8e6","Type":"ContainerStarted","Data":"d80b2fdc62f60d2d25cc7c426407eec80ec1fedfb57f6e806624299d38dc9a14"} Oct 31 00:55:55.864458 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.864309 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-39.us-east-2.compute.internal" event={"ID":"eaafb8f2be56642836cc3cf93864aa97","Type":"ContainerStarted","Data":"e6a5b6c9a5074780e347cae94958840fdae61b0458a8fb76adbcbc4c166efbff"} Oct 31 00:55:55.864458 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.864322 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-39.us-east-2.compute.internal" event={"ID":"eaafb8f2be56642836cc3cf93864aa97","Type":"ContainerStarted","Data":"cd31ba38051bd98153d6964a968a80f77392c88ca1a73ad450f84bf749ba8b49"} Oct 31 00:55:55.864458 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.864322 4440 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal" Oct 31 00:55:55.864458 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.864438 4440 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-39.us-east-2.compute.internal" Oct 31 00:55:55.888320 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.888285 4440 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:55:55.888473 ip-10-0-130-39 kubenswrapper[4440]: E1031 00:55:55.888343 4440 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-39.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-39.us-east-2.compute.internal" Oct 31 00:55:55.889705 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.889684 4440 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:55:55.889798 ip-10-0-130-39 kubenswrapper[4440]: E1031 00:55:55.889732 4440 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal" Oct 31 00:55:55.931875 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.931843 4440 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:55:55.933155 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.933134 4440 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-39.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:55:55.933261 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.933173 4440 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-39.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:55:55.933261 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.933184 4440 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-39.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:55:55.933261 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.933215 4440 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-39.us-east-2.compute.internal" Oct 31 00:55:55.945851 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.945820 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ce6a902de7718819f66be8130076b8e6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal\" (UID: \"ce6a902de7718819f66be8130076b8e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal" Oct 31 00:55:55.945912 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.945864 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce6a902de7718819f66be8130076b8e6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal\" (UID: \"ce6a902de7718819f66be8130076b8e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal" Oct 31 00:55:55.945912 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.945882 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eaafb8f2be56642836cc3cf93864aa97-config\") pod \"kube-apiserver-proxy-ip-10-0-130-39.us-east-2.compute.internal\" (UID: \"eaafb8f2be56642836cc3cf93864aa97\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-39.us-east-2.compute.internal" Oct 31 00:55:55.951064 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.951047 4440 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-130-39.us-east-2.compute.internal" Oct 31 00:55:55.951116 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:55.951108 4440 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-39.us-east-2.compute.internal" Oct 31 00:55:56.046258 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.046212 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ce6a902de7718819f66be8130076b8e6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal\" (UID: \"ce6a902de7718819f66be8130076b8e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal" Oct 31 00:55:56.046457 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.046277 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce6a902de7718819f66be8130076b8e6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal\" (UID: \"ce6a902de7718819f66be8130076b8e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal" Oct 31 00:55:56.046457 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.046299 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eaafb8f2be56642836cc3cf93864aa97-config\") pod \"kube-apiserver-proxy-ip-10-0-130-39.us-east-2.compute.internal\" (UID: \"eaafb8f2be56642836cc3cf93864aa97\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-39.us-east-2.compute.internal" Oct 31 00:55:56.046457 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.046234 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ce6a902de7718819f66be8130076b8e6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal\" (UID: \"ce6a902de7718819f66be8130076b8e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal" Oct 31 00:55:56.046457 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.046339 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eaafb8f2be56642836cc3cf93864aa97-config\") pod \"kube-apiserver-proxy-ip-10-0-130-39.us-east-2.compute.internal\" (UID: \"eaafb8f2be56642836cc3cf93864aa97\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-39.us-east-2.compute.internal" Oct 31 00:55:56.046457 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.046362 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce6a902de7718819f66be8130076b8e6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal\" (UID: \"ce6a902de7718819f66be8130076b8e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal" Oct 31 00:55:56.718307 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.718269 4440 apiserver.go:52] "Watching apiserver" Oct 31 00:55:56.725468 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.725445 4440 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 00:55:56.725874 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.725853 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-sk9vt","openshift-dns/dns-default-zhmhj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-p8vd6","openshift-cluster-node-tuning-operator/tuned-8r7qf","openshift-image-registry/node-ca-qzm2t","openshift-ovn-kubernetes/ovnkube-node-vcdmb","openshift-multus/network-metrics-daemon-pcvbh","openshift-network-operator/iptables-alerter-mtwcc","kube-system/konnectivity-agent-8kln5","openshift-dns/node-resolver-qsppl","openshift-kube-storage-version-migrator/migrator-66df5d496b-lb9mz","kube-system/kube-apiserver-proxy-ip-10-0-130-39.us-east-2.compute.internal","openshift-network-diagnostics/network-check-target-5xvng","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg","openshift-multus/multus-g5h4s","kube-system/global-pull-secret-syncer-s9tlh","openshift-ingress-canary/ingress-canary-srw8h"] Oct 31 00:55:56.731093 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.731058 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:55:56.731221 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.731171 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 00:55:56.731221 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.731184 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-xb7hg\"" Oct 31 00:55:56.731221 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.731186 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 00:55:56.731356 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.731233 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-89rf9\"" Oct 31 00:55:56.731356 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.731243 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 00:55:56.731579 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.731564 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9dpj5\"" Oct 31 00:55:56.731648 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.731594 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 00:55:56.731648 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.731611 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 00:55:56.732019 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.732003 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pcvbh" Oct 31 00:55:56.732375 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.732356 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:55:56.732482 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.732406 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-j9qsc\"" Oct 31 00:55:56.732838 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.732825 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 00:55:56.732838 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.732833 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-m2wnh\"" Oct 31 00:55:56.732959 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.732947 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 00:55:56.734211 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.734189 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 00:55:56.734358 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.734333 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 00:55:56.734461 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.734367 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 00:55:56.734461 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.734407 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4j5v2\"" Oct 31 00:55:56.734544 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.734499 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xcm4p\"" Oct 31 00:55:56.734880 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.734860 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zhmhj" Oct 31 00:55:56.735074 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.735052 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 00:55:56.735074 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.735067 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 00:55:56.735237 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.735222 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 00:55:56.737030 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.736999 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 00:55:56.737030 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.737011 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 00:55:56.737149 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.737094 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 00:55:56.737228 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.737212 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 00:55:56.737268 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.737217 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 00:55:56.737268 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.737240 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 00:55:56.737268 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.737254 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 00:55:56.737450 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.737298 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-89pfs\"" Oct 31 00:55:56.737450 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.737393 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 00:55:56.737543 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.737461 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 00:55:56.737543 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.737513 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 00:55:56.738753 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.738737 4440 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 00:55:56.744752 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.744733 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 00:55:56.744969 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.744957 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-xxmpd\"" Oct 31 00:55:56.745635 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.745619 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 00:55:56.745862 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.745848 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 00:55:56.746274 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.746259 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-xg6v4\"" Oct 31 00:55:56.746335 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.746320 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-72bs2\"" Oct 31 00:55:56.746562 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.746547 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-gczx4\"" Oct 31 00:55:56.746627 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.746614 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-ht5gb\"" Oct 31 00:55:56.747142 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.747119 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 00:55:56.747210 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.747166 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 00:55:56.747210 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.747201 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 00:55:56.747323 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.747238 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 31 00:55:56.747323 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.747255 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 00:55:56.747323 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.747289 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 31 00:55:56.747323 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.747304 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 00:55:56.747529 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.747341 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 31 00:55:56.747529 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.747441 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 00:55:56.747628 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.747534 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 00:55:56.747628 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.747567 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 31 00:55:56.747628 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.747579 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 31 00:55:56.747628 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.747586 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 31 00:55:56.747628 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.747610 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-jrg45\"" Oct 31 00:55:56.747779 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.747702 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-dlpqk\"" Oct 31 00:55:56.747779 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.747713 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-58wnq\"" Oct 31 00:55:56.748130 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.748117 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 00:55:56.749184 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.749165 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd654994-98a0-4d39-aa38-270b4a0d4341-tmp-dir\") pod \"node-resolver-qsppl\" (UID: \"cd654994-98a0-4d39-aa38-270b4a0d4341\") " pod="openshift-dns/node-resolver-qsppl" Oct 31 00:55:56.749262 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.749204 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c3939885-f5d4-4bd2-a69f-435c2a8fa048-etc-selinux\") pod \"aws-ebs-csi-driver-node-cq2wg\" (UID: \"c3939885-f5d4-4bd2-a69f-435c2a8fa048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:56.749262 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.749236 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f-metrics-certs\") pod \"network-metrics-daemon-pcvbh\" (UID: \"0fe8107c-9128-4c8a-9bde-e1cacd2ed19f\") " pod="openshift-multus/network-metrics-daemon-pcvbh" Oct 31 00:55:56.749351 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.749263 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-ovn-node-metrics-cert\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.749351 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.749317 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd654994-98a0-4d39-aa38-270b4a0d4341-tmp-dir\") pod \"node-resolver-qsppl\" (UID: \"cd654994-98a0-4d39-aa38-270b4a0d4341\") " pod="openshift-dns/node-resolver-qsppl" Oct 31 00:55:56.749351 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.749326 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdbsq\" (UniqueName: \"kubernetes.io/projected/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-kube-api-access-fdbsq\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.749534 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.749372 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c3939885-f5d4-4bd2-a69f-435c2a8fa048-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cq2wg\" (UID: \"c3939885-f5d4-4bd2-a69f-435c2a8fa048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:56.749534 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.749402 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6ddfc5f-887b-4821-bac9-54cb59c50e32-tmp\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:56.749534 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.749447 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-node-log\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.749534 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.749478 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-host-cni-netd\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.749534 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.749504 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-host-run-netns\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.749534 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.749529 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6ddfc5f-887b-4821-bac9-54cb59c50e32-tmp\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:56.749825 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.749569 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs6kv\" (UniqueName: \"kubernetes.io/projected/d610021a-d4d9-473d-8889-cb5afcb5b8a0-kube-api-access-xs6kv\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.749825 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.749599 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j6vc\" (UniqueName: \"kubernetes.io/projected/c3939885-f5d4-4bd2-a69f-435c2a8fa048-kube-api-access-4j6vc\") pod \"aws-ebs-csi-driver-node-cq2wg\" (UID: \"c3939885-f5d4-4bd2-a69f-435c2a8fa048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:56.749825 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.749617 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9v2m\" (UniqueName: \"kubernetes.io/projected/4b8ba0db-d8b9-4a40-a144-1400634cd99d-kube-api-access-t9v2m\") pod \"iptables-alerter-mtwcc\" (UID: \"4b8ba0db-d8b9-4a40-a144-1400634cd99d\") " pod="openshift-network-operator/iptables-alerter-mtwcc" Oct 31 00:55:56.749825 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.749647 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-sys\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:56.749825 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.749672 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a11b541-6b88-439e-bcbe-5901cb4c20b0-cert\") pod \"ingress-canary-srw8h\" (UID: \"9a11b541-6b88-439e-bcbe-5901cb4c20b0\") " pod="openshift-ingress-canary/ingress-canary-srw8h" Oct 31 00:55:56.749825 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.749691 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-system-cni-dir\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.749825 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.749709 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-multus-conf-dir\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.749825 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.749725 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4qw7\" (UniqueName: \"kubernetes.io/projected/cd654994-98a0-4d39-aa38-270b4a0d4341-kube-api-access-m4qw7\") pod \"node-resolver-qsppl\" (UID: \"cd654994-98a0-4d39-aa38-270b4a0d4341\") " pod="openshift-dns/node-resolver-qsppl" Oct 31 00:55:56.749825 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.749739 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b8ba0db-d8b9-4a40-a144-1400634cd99d-host-slash\") pod \"iptables-alerter-mtwcc\" (UID: \"4b8ba0db-d8b9-4a40-a144-1400634cd99d\") " pod="openshift-network-operator/iptables-alerter-mtwcc" Oct 31 00:55:56.749825 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.749756 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-etc-modprobe-d\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:56.749825 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.749771 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-etc-systemd\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:56.750311 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.750195 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-ovn-node-metrics-cert\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.750311 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.750240 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/88c1a03a-cb94-4226-9d53-cd923895569e-cni-binary-copy\") pod \"multus-additional-cni-plugins-p8vd6\" (UID: \"88c1a03a-cb94-4226-9d53-cd923895569e\") " pod="openshift-multus/multus-additional-cni-plugins-p8vd6" Oct 31 00:55:56.750412 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.750335 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a11b541-6b88-439e-bcbe-5901cb4c20b0-cert\") pod \"ingress-canary-srw8h\" (UID: \"9a11b541-6b88-439e-bcbe-5901cb4c20b0\") " pod="openshift-ingress-canary/ingress-canary-srw8h" Oct 31 00:55:56.750720 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.750706 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ee8bc72c-f181-452d-ab71-b13a51672bcf-signing-cabundle\") pod \"service-ca-5fffc97bdf-sk9vt\" (UID: \"ee8bc72c-f181-452d-ab71-b13a51672bcf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sk9vt" Oct 31 00:55:56.750818 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.750803 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-host-var-lib-cni-bin\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.750879 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.750860 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-host-var-lib-kubelet\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.750920 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.750900 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cd654994-98a0-4d39-aa38-270b4a0d4341-hosts-file\") pod \"node-resolver-qsppl\" (UID: \"cd654994-98a0-4d39-aa38-270b4a0d4341\") " pod="openshift-dns/node-resolver-qsppl" Oct 31 00:55:56.750966 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.750925 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwrz5\" (UniqueName: \"kubernetes.io/projected/3c9845ea-a9d6-4d5c-93ea-fb096666a2b9-kube-api-access-kwrz5\") pod \"network-check-target-5xvng\" (UID: \"3c9845ea-a9d6-4d5c-93ea-fb096666a2b9\") " pod="openshift-network-diagnostics/network-check-target-5xvng" Oct 31 00:55:56.751114 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.750974 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ee8bc72c-f181-452d-ab71-b13a51672bcf-signing-cabundle\") pod \"service-ca-5fffc97bdf-sk9vt\" (UID: \"ee8bc72c-f181-452d-ab71-b13a51672bcf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sk9vt" Oct 31 00:55:56.751114 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.750976 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/12002bf6-fce5-4d20-a62c-6782522acd47-konnectivity-ca\") pod \"konnectivity-agent-8kln5\" (UID: \"12002bf6-fce5-4d20-a62c-6782522acd47\") " pod="kube-system/konnectivity-agent-8kln5" Oct 31 00:55:56.751114 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.750991 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/88c1a03a-cb94-4226-9d53-cd923895569e-cni-binary-copy\") pod \"multus-additional-cni-plugins-p8vd6\" (UID: \"88c1a03a-cb94-4226-9d53-cd923895569e\") " pod="openshift-multus/multus-additional-cni-plugins-p8vd6" Oct 31 00:55:56.751114 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.751019 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzddr\" (UniqueName: \"kubernetes.io/projected/49f7c8f4-b979-436d-b035-20f9fcae140a-kube-api-access-vzddr\") pod \"dns-default-zhmhj\" (UID: \"49f7c8f4-b979-436d-b035-20f9fcae140a\") " pod="openshift-dns/dns-default-zhmhj" Oct 31 00:55:56.751114 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.751045 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d610021a-d4d9-473d-8889-cb5afcb5b8a0-cni-binary-copy\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.751114 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.751086 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-host-run-k8s-cni-cncf-io\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.751114 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.751104 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-var-lib-kubelet\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:56.751487 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.751207 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49f7c8f4-b979-436d-b035-20f9fcae140a-tmp-dir\") pod \"dns-default-zhmhj\" (UID: \"49f7c8f4-b979-436d-b035-20f9fcae140a\") " pod="openshift-dns/dns-default-zhmhj" Oct 31 00:55:56.751487 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.751281 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/49f8a438-6b29-4160-98e1-3f26e27fe0af-kubelet-config\") pod \"global-pull-secret-syncer-s9tlh\" (UID: \"49f8a438-6b29-4160-98e1-3f26e27fe0af\") " pod="kube-system/global-pull-secret-syncer-s9tlh" Oct 31 00:55:56.751487 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.751340 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49f7c8f4-b979-436d-b035-20f9fcae140a-tmp-dir\") pod \"dns-default-zhmhj\" (UID: \"49f7c8f4-b979-436d-b035-20f9fcae140a\") " pod="openshift-dns/dns-default-zhmhj" Oct 31 00:55:56.751487 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.751358 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/12002bf6-fce5-4d20-a62c-6782522acd47-konnectivity-ca\") pod \"konnectivity-agent-8kln5\" (UID: \"12002bf6-fce5-4d20-a62c-6782522acd47\") " pod="kube-system/konnectivity-agent-8kln5" Oct 31 00:55:56.751487 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.751379 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d610021a-d4d9-473d-8889-cb5afcb5b8a0-cni-binary-copy\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.751487 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.751385 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/49f8a438-6b29-4160-98e1-3f26e27fe0af-original-pull-secret\") pod \"global-pull-secret-syncer-s9tlh\" (UID: \"49f8a438-6b29-4160-98e1-3f26e27fe0af\") " pod="kube-system/global-pull-secret-syncer-s9tlh" Oct 31 00:55:56.751721 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.751506 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88c1a03a-cb94-4226-9d53-cd923895569e-system-cni-dir\") pod \"multus-additional-cni-plugins-p8vd6\" (UID: \"88c1a03a-cb94-4226-9d53-cd923895569e\") " pod="openshift-multus/multus-additional-cni-plugins-p8vd6" Oct 31 00:55:56.751721 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.751549 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-host-kubelet\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.751721 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.751595 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-run-systemd\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.751721 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.751617 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-multus-cni-dir\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.751721 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.751632 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-run\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:56.751721 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.751631 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/49f8a438-6b29-4160-98e1-3f26e27fe0af-original-pull-secret\") pod \"global-pull-secret-syncer-s9tlh\" (UID: \"49f8a438-6b29-4160-98e1-3f26e27fe0af\") " pod="kube-system/global-pull-secret-syncer-s9tlh" Oct 31 00:55:56.751721 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.751659 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a6ddfc5f-887b-4821-bac9-54cb59c50e32-etc-tuned\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:56.751721 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.751702 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-host-slash\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.752032 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.751729 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-var-lib-openvswitch\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.752032 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.751759 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-env-overrides\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.752032 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.751773 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a6ddfc5f-887b-4821-bac9-54cb59c50e32-etc-tuned\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:56.752032 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.751806 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a37f05b-0755-4ca7-8d9b-12030234bec1-host\") pod \"node-ca-qzm2t\" (UID: \"0a37f05b-0755-4ca7-8d9b-12030234bec1\") " pod="openshift-image-registry/node-ca-qzm2t" Oct 31 00:55:56.752032 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.751826 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4b8ba0db-d8b9-4a40-a144-1400634cd99d-iptables-alerter-script\") pod \"iptables-alerter-mtwcc\" (UID: \"4b8ba0db-d8b9-4a40-a144-1400634cd99d\") " pod="openshift-network-operator/iptables-alerter-mtwcc" Oct 31 00:55:56.752032 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.751843 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnvxk\" (UniqueName: \"kubernetes.io/projected/a6ddfc5f-887b-4821-bac9-54cb59c50e32-kube-api-access-fnvxk\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:56.752032 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.751897 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnc9j\" (UniqueName: \"kubernetes.io/projected/9a11b541-6b88-439e-bcbe-5901cb4c20b0-kube-api-access-hnc9j\") pod \"ingress-canary-srw8h\" (UID: \"9a11b541-6b88-439e-bcbe-5901cb4c20b0\") " pod="openshift-ingress-canary/ingress-canary-srw8h" Oct 31 00:55:56.752032 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.751938 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-env-overrides\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.752362 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.752055 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klvmv\" (UniqueName: \"kubernetes.io/projected/ee8bc72c-f181-452d-ab71-b13a51672bcf-kube-api-access-klvmv\") pod \"service-ca-5fffc97bdf-sk9vt\" (UID: \"ee8bc72c-f181-452d-ab71-b13a51672bcf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sk9vt" Oct 31 00:55:56.752362 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.752050 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4b8ba0db-d8b9-4a40-a144-1400634cd99d-iptables-alerter-script\") pod \"iptables-alerter-mtwcc\" (UID: \"4b8ba0db-d8b9-4a40-a144-1400634cd99d\") " pod="openshift-network-operator/iptables-alerter-mtwcc" Oct 31 00:55:56.752362 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.752077 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/49f8a438-6b29-4160-98e1-3f26e27fe0af-dbus\") pod \"global-pull-secret-syncer-s9tlh\" (UID: \"49f8a438-6b29-4160-98e1-3f26e27fe0af\") " pod="kube-system/global-pull-secret-syncer-s9tlh" Oct 31 00:55:56.752362 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.752095 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-multus-socket-dir-parent\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.752362 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.752120 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c3939885-f5d4-4bd2-a69f-435c2a8fa048-sys-fs\") pod \"aws-ebs-csi-driver-node-cq2wg\" (UID: \"c3939885-f5d4-4bd2-a69f-435c2a8fa048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:56.752362 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.752162 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ee8bc72c-f181-452d-ab71-b13a51672bcf-signing-key\") pod \"service-ca-5fffc97bdf-sk9vt\" (UID: \"ee8bc72c-f181-452d-ab71-b13a51672bcf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sk9vt" Oct 31 00:55:56.752362 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.752183 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nchj\" (UniqueName: \"kubernetes.io/projected/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f-kube-api-access-7nchj\") pod \"network-metrics-daemon-pcvbh\" (UID: \"0fe8107c-9128-4c8a-9bde-e1cacd2ed19f\") " pod="openshift-multus/network-metrics-daemon-pcvbh" Oct 31 00:55:56.752362 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.752205 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-host-run-multus-certs\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.752362 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.752236 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0a37f05b-0755-4ca7-8d9b-12030234bec1-serviceca\") pod \"node-ca-qzm2t\" (UID: \"0a37f05b-0755-4ca7-8d9b-12030234bec1\") " pod="openshift-image-registry/node-ca-qzm2t" Oct 31 00:55:56.752825 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.752668 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-etc-kubernetes\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:56.752825 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.752723 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-etc-openvswitch\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.752825 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.752751 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c3939885-f5d4-4bd2-a69f-435c2a8fa048-socket-dir\") pod \"aws-ebs-csi-driver-node-cq2wg\" (UID: \"c3939885-f5d4-4bd2-a69f-435c2a8fa048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:56.752825 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.752778 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0a37f05b-0755-4ca7-8d9b-12030234bec1-serviceca\") pod \"node-ca-qzm2t\" (UID: \"0a37f05b-0755-4ca7-8d9b-12030234bec1\") " pod="openshift-image-registry/node-ca-qzm2t" Oct 31 00:55:56.752825 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.752812 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ee8bc72c-f181-452d-ab71-b13a51672bcf-signing-key\") pod \"service-ca-5fffc97bdf-sk9vt\" (UID: \"ee8bc72c-f181-452d-ab71-b13a51672bcf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sk9vt" Oct 31 00:55:56.753034 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.752841 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-host\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:56.753034 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.752922 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/88c1a03a-cb94-4226-9d53-cd923895569e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p8vd6\" (UID: \"88c1a03a-cb94-4226-9d53-cd923895569e\") " pod="openshift-multus/multus-additional-cni-plugins-p8vd6" Oct 31 00:55:56.753034 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.752962 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/88c1a03a-cb94-4226-9d53-cd923895569e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p8vd6\" (UID: \"88c1a03a-cb94-4226-9d53-cd923895569e\") " pod="openshift-multus/multus-additional-cni-plugins-p8vd6" Oct 31 00:55:56.753034 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.753009 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/88c1a03a-cb94-4226-9d53-cd923895569e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p8vd6\" (UID: \"88c1a03a-cb94-4226-9d53-cd923895569e\") " pod="openshift-multus/multus-additional-cni-plugins-p8vd6" Oct 31 00:55:56.753212 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.753053 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-986bh\" (UniqueName: \"kubernetes.io/projected/88c1a03a-cb94-4226-9d53-cd923895569e-kube-api-access-986bh\") pod \"multus-additional-cni-plugins-p8vd6\" (UID: \"88c1a03a-cb94-4226-9d53-cd923895569e\") " pod="openshift-multus/multus-additional-cni-plugins-p8vd6" Oct 31 00:55:56.753212 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.753079 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-systemd-units\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.753212 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.753111 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-run-openvswitch\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.753212 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.753162 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-cnibin\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.753212 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.753206 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d610021a-d4d9-473d-8889-cb5afcb5b8a0-multus-daemon-config\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.753446 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.753238 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-ovnkube-script-lib\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.753446 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.753281 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npk86\" (UniqueName: \"kubernetes.io/projected/0a37f05b-0755-4ca7-8d9b-12030234bec1-kube-api-access-npk86\") pod \"node-ca-qzm2t\" (UID: \"0a37f05b-0755-4ca7-8d9b-12030234bec1\") " pod="openshift-image-registry/node-ca-qzm2t" Oct 31 00:55:56.753446 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.753308 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-etc-sysconfig\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:56.753446 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.753336 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-etc-sysctl-d\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:56.753446 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.753374 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-lib-modules\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:56.753446 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.753405 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/12002bf6-fce5-4d20-a62c-6782522acd47-agent-certs\") pod \"konnectivity-agent-8kln5\" (UID: \"12002bf6-fce5-4d20-a62c-6782522acd47\") " pod="kube-system/konnectivity-agent-8kln5" Oct 31 00:55:56.753700 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.753457 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49f7c8f4-b979-436d-b035-20f9fcae140a-config-volume\") pod \"dns-default-zhmhj\" (UID: \"49f7c8f4-b979-436d-b035-20f9fcae140a\") " pod="openshift-dns/dns-default-zhmhj" Oct 31 00:55:56.753700 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.753484 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfhcf\" (UniqueName: \"kubernetes.io/projected/ac0f8a44-5a66-40fe-a08c-080314e7ea60-kube-api-access-hfhcf\") pod \"migrator-66df5d496b-lb9mz\" (UID: \"ac0f8a44-5a66-40fe-a08c-080314e7ea60\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lb9mz" Oct 31 00:55:56.753700 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.753527 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-hostroot\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.753700 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.753562 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-etc-kubernetes\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.753700 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.753602 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/88c1a03a-cb94-4226-9d53-cd923895569e-cnibin\") pod \"multus-additional-cni-plugins-p8vd6\" (UID: \"88c1a03a-cb94-4226-9d53-cd923895569e\") " pod="openshift-multus/multus-additional-cni-plugins-p8vd6" Oct 31 00:55:56.753700 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.753626 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-run-ovn\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.753700 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.753667 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-log-socket\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.753700 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.753696 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-host-run-ovn-kubernetes\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.754025 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.753738 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-ovnkube-config\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.754025 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.753772 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c3939885-f5d4-4bd2-a69f-435c2a8fa048-device-dir\") pod \"aws-ebs-csi-driver-node-cq2wg\" (UID: \"c3939885-f5d4-4bd2-a69f-435c2a8fa048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:56.754025 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.753801 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-etc-sysctl-conf\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:56.754025 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.753842 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49f7c8f4-b979-436d-b035-20f9fcae140a-metrics-tls\") pod \"dns-default-zhmhj\" (UID: \"49f7c8f4-b979-436d-b035-20f9fcae140a\") " pod="openshift-dns/dns-default-zhmhj" Oct 31 00:55:56.754025 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.753865 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-host-run-netns\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.754025 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.753905 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-host-cni-bin\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.754025 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.753934 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.754025 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.753978 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-host-var-lib-cni-multus\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.754025 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.754006 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c3939885-f5d4-4bd2-a69f-435c2a8fa048-registration-dir\") pod \"aws-ebs-csi-driver-node-cq2wg\" (UID: \"c3939885-f5d4-4bd2-a69f-435c2a8fa048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:56.754444 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.754390 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/88c1a03a-cb94-4226-9d53-cd923895569e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p8vd6\" (UID: \"88c1a03a-cb94-4226-9d53-cd923895569e\") " pod="openshift-multus/multus-additional-cni-plugins-p8vd6" Oct 31 00:55:56.754644 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.754625 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/88c1a03a-cb94-4226-9d53-cd923895569e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p8vd6\" (UID: \"88c1a03a-cb94-4226-9d53-cd923895569e\") " pod="openshift-multus/multus-additional-cni-plugins-p8vd6" Oct 31 00:55:56.755154 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.755133 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d610021a-d4d9-473d-8889-cb5afcb5b8a0-multus-daemon-config\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.755376 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.755354 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-ovnkube-script-lib\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.755870 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.755849 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49f7c8f4-b979-436d-b035-20f9fcae140a-config-volume\") pod \"dns-default-zhmhj\" (UID: \"49f7c8f4-b979-436d-b035-20f9fcae140a\") " pod="openshift-dns/dns-default-zhmhj" Oct 31 00:55:56.755976 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.755898 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/12002bf6-fce5-4d20-a62c-6782522acd47-agent-certs\") pod \"konnectivity-agent-8kln5\" (UID: \"12002bf6-fce5-4d20-a62c-6782522acd47\") " pod="kube-system/konnectivity-agent-8kln5" Oct 31 00:55:56.756035 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.755997 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-ovnkube-config\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.757708 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.757688 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwrz5\" (UniqueName: \"kubernetes.io/projected/3c9845ea-a9d6-4d5c-93ea-fb096666a2b9-kube-api-access-kwrz5\") pod \"network-check-target-5xvng\" (UID: \"3c9845ea-a9d6-4d5c-93ea-fb096666a2b9\") " pod="openshift-network-diagnostics/network-check-target-5xvng" Oct 31 00:55:56.757708 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.757699 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzddr\" (UniqueName: \"kubernetes.io/projected/49f7c8f4-b979-436d-b035-20f9fcae140a-kube-api-access-vzddr\") pod \"dns-default-zhmhj\" (UID: \"49f7c8f4-b979-436d-b035-20f9fcae140a\") " pod="openshift-dns/dns-default-zhmhj" Oct 31 00:55:56.757708 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.757724 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs6kv\" (UniqueName: \"kubernetes.io/projected/d610021a-d4d9-473d-8889-cb5afcb5b8a0-kube-api-access-xs6kv\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.757870 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.757750 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdbsq\" (UniqueName: \"kubernetes.io/projected/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-kube-api-access-fdbsq\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.757870 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.757820 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4qw7\" (UniqueName: \"kubernetes.io/projected/cd654994-98a0-4d39-aa38-270b4a0d4341-kube-api-access-m4qw7\") pod \"node-resolver-qsppl\" (UID: \"cd654994-98a0-4d39-aa38-270b4a0d4341\") " pod="openshift-dns/node-resolver-qsppl" Oct 31 00:55:56.757870 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.757854 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j6vc\" (UniqueName: \"kubernetes.io/projected/c3939885-f5d4-4bd2-a69f-435c2a8fa048-kube-api-access-4j6vc\") pod \"aws-ebs-csi-driver-node-cq2wg\" (UID: \"c3939885-f5d4-4bd2-a69f-435c2a8fa048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:56.757953 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.757854 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9v2m\" (UniqueName: \"kubernetes.io/projected/4b8ba0db-d8b9-4a40-a144-1400634cd99d-kube-api-access-t9v2m\") pod \"iptables-alerter-mtwcc\" (UID: \"4b8ba0db-d8b9-4a40-a144-1400634cd99d\") " pod="openshift-network-operator/iptables-alerter-mtwcc" Oct 31 00:55:56.758850 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.758830 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klvmv\" (UniqueName: \"kubernetes.io/projected/ee8bc72c-f181-452d-ab71-b13a51672bcf-kube-api-access-klvmv\") pod \"service-ca-5fffc97bdf-sk9vt\" (UID: \"ee8bc72c-f181-452d-ab71-b13a51672bcf\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sk9vt" Oct 31 00:55:56.759186 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.759167 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnc9j\" (UniqueName: \"kubernetes.io/projected/9a11b541-6b88-439e-bcbe-5901cb4c20b0-kube-api-access-hnc9j\") pod \"ingress-canary-srw8h\" (UID: \"9a11b541-6b88-439e-bcbe-5901cb4c20b0\") " pod="openshift-ingress-canary/ingress-canary-srw8h" Oct 31 00:55:56.759373 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.759359 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnvxk\" (UniqueName: \"kubernetes.io/projected/a6ddfc5f-887b-4821-bac9-54cb59c50e32-kube-api-access-fnvxk\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:56.759584 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.759569 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nchj\" (UniqueName: \"kubernetes.io/projected/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f-kube-api-access-7nchj\") pod \"network-metrics-daemon-pcvbh\" (UID: \"0fe8107c-9128-4c8a-9bde-e1cacd2ed19f\") " pod="openshift-multus/network-metrics-daemon-pcvbh" Oct 31 00:55:56.760433 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.760394 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-986bh\" (UniqueName: \"kubernetes.io/projected/88c1a03a-cb94-4226-9d53-cd923895569e-kube-api-access-986bh\") pod \"multus-additional-cni-plugins-p8vd6\" (UID: \"88c1a03a-cb94-4226-9d53-cd923895569e\") " pod="openshift-multus/multus-additional-cni-plugins-p8vd6" Oct 31 00:55:56.760887 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.760869 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npk86\" (UniqueName: \"kubernetes.io/projected/0a37f05b-0755-4ca7-8d9b-12030234bec1-kube-api-access-npk86\") pod \"node-ca-qzm2t\" (UID: \"0a37f05b-0755-4ca7-8d9b-12030234bec1\") " pod="openshift-image-registry/node-ca-qzm2t" Oct 31 00:55:56.761258 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.761243 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfhcf\" (UniqueName: \"kubernetes.io/projected/ac0f8a44-5a66-40fe-a08c-080314e7ea60-kube-api-access-hfhcf\") pod \"migrator-66df5d496b-lb9mz\" (UID: \"ac0f8a44-5a66-40fe-a08c-080314e7ea60\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lb9mz" Oct 31 00:55:56.801378 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.801343 4440 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal" Oct 31 00:55:56.801566 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.801469 4440 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-39.us-east-2.compute.internal" Oct 31 00:55:56.813333 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.813308 4440 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:55:56.813517 ip-10-0-130-39 kubenswrapper[4440]: E1031 00:55:56.813351 4440 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-39.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-39.us-east-2.compute.internal" Oct 31 00:55:56.813742 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.813726 4440 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:55:56.813781 ip-10-0-130-39 kubenswrapper[4440]: E1031 00:55:56.813764 4440 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-39.us-east-2.compute.internal" Oct 31 00:55:56.854982 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.854951 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-run\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:56.854982 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.854989 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-host-slash\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.855243 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855007 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-var-lib-openvswitch\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.855243 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855031 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a37f05b-0755-4ca7-8d9b-12030234bec1-host\") pod \"node-ca-qzm2t\" (UID: \"0a37f05b-0755-4ca7-8d9b-12030234bec1\") " pod="openshift-image-registry/node-ca-qzm2t" Oct 31 00:55:56.855243 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855056 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/49f8a438-6b29-4160-98e1-3f26e27fe0af-dbus\") pod \"global-pull-secret-syncer-s9tlh\" (UID: \"49f8a438-6b29-4160-98e1-3f26e27fe0af\") " pod="kube-system/global-pull-secret-syncer-s9tlh" Oct 31 00:55:56.855243 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855079 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-multus-socket-dir-parent\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.855243 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855090 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-host-slash\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.855243 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855104 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c3939885-f5d4-4bd2-a69f-435c2a8fa048-sys-fs\") pod \"aws-ebs-csi-driver-node-cq2wg\" (UID: \"c3939885-f5d4-4bd2-a69f-435c2a8fa048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:56.855243 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855092 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-run\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:56.855243 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855113 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a37f05b-0755-4ca7-8d9b-12030234bec1-host\") pod \"node-ca-qzm2t\" (UID: \"0a37f05b-0755-4ca7-8d9b-12030234bec1\") " pod="openshift-image-registry/node-ca-qzm2t" Oct 31 00:55:56.855243 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855092 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-var-lib-openvswitch\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.855243 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855163 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c3939885-f5d4-4bd2-a69f-435c2a8fa048-sys-fs\") pod \"aws-ebs-csi-driver-node-cq2wg\" (UID: \"c3939885-f5d4-4bd2-a69f-435c2a8fa048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:56.855243 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855164 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/49f8a438-6b29-4160-98e1-3f26e27fe0af-dbus\") pod \"global-pull-secret-syncer-s9tlh\" (UID: \"49f8a438-6b29-4160-98e1-3f26e27fe0af\") " pod="kube-system/global-pull-secret-syncer-s9tlh" Oct 31 00:55:56.855243 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855175 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-host-run-multus-certs\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.855243 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855202 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-multus-socket-dir-parent\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.855243 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855216 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-host-run-multus-certs\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.855243 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855219 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-etc-kubernetes\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855267 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-etc-kubernetes\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855272 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-etc-openvswitch\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855298 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c3939885-f5d4-4bd2-a69f-435c2a8fa048-socket-dir\") pod \"aws-ebs-csi-driver-node-cq2wg\" (UID: \"c3939885-f5d4-4bd2-a69f-435c2a8fa048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855314 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-host\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855322 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-etc-openvswitch\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855330 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/88c1a03a-cb94-4226-9d53-cd923895569e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p8vd6\" (UID: \"88c1a03a-cb94-4226-9d53-cd923895569e\") " pod="openshift-multus/multus-additional-cni-plugins-p8vd6" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855356 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-systemd-units\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855366 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c3939885-f5d4-4bd2-a69f-435c2a8fa048-socket-dir\") pod \"aws-ebs-csi-driver-node-cq2wg\" (UID: \"c3939885-f5d4-4bd2-a69f-435c2a8fa048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855379 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-run-openvswitch\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855384 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-host\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855402 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-systemd-units\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855405 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-cnibin\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855412 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/88c1a03a-cb94-4226-9d53-cd923895569e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p8vd6\" (UID: \"88c1a03a-cb94-4226-9d53-cd923895569e\") " pod="openshift-multus/multus-additional-cni-plugins-p8vd6" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855443 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-run-openvswitch\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855462 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-etc-sysconfig\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855476 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-cnibin\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855481 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-etc-sysctl-d\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855496 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-lib-modules\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855513 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-hostroot\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855541 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-hostroot\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855556 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-etc-kubernetes\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855565 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-etc-sysconfig\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855570 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/88c1a03a-cb94-4226-9d53-cd923895569e-cnibin\") pod \"multus-additional-cni-plugins-p8vd6\" (UID: \"88c1a03a-cb94-4226-9d53-cd923895569e\") " pod="openshift-multus/multus-additional-cni-plugins-p8vd6" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855586 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/88c1a03a-cb94-4226-9d53-cd923895569e-cnibin\") pod \"multus-additional-cni-plugins-p8vd6\" (UID: \"88c1a03a-cb94-4226-9d53-cd923895569e\") " pod="openshift-multus/multus-additional-cni-plugins-p8vd6" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855610 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-lib-modules\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855608 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-run-ovn\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855616 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-etc-kubernetes\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855633 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-log-socket\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855640 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-run-ovn\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855648 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-host-run-ovn-kubernetes\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855670 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c3939885-f5d4-4bd2-a69f-435c2a8fa048-device-dir\") pod \"aws-ebs-csi-driver-node-cq2wg\" (UID: \"c3939885-f5d4-4bd2-a69f-435c2a8fa048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855692 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-etc-sysctl-conf\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855695 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-host-run-ovn-kubernetes\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855702 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c3939885-f5d4-4bd2-a69f-435c2a8fa048-device-dir\") pod \"aws-ebs-csi-driver-node-cq2wg\" (UID: \"c3939885-f5d4-4bd2-a69f-435c2a8fa048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855674 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-log-socket\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855716 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49f7c8f4-b979-436d-b035-20f9fcae140a-metrics-tls\") pod \"dns-default-zhmhj\" (UID: \"49f7c8f4-b979-436d-b035-20f9fcae140a\") " pod="openshift-dns/dns-default-zhmhj" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855737 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-host-run-netns\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855751 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-host-cni-bin\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855767 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855784 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-etc-sysctl-conf\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855787 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-host-var-lib-cni-multus\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855795 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-host-run-netns\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855711 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-etc-sysctl-d\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855822 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c3939885-f5d4-4bd2-a69f-435c2a8fa048-registration-dir\") pod \"aws-ebs-csi-driver-node-cq2wg\" (UID: \"c3939885-f5d4-4bd2-a69f-435c2a8fa048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: E1031 00:55:56.855816 4440 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855839 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-host-var-lib-cni-multus\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855832 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-host-cni-bin\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855850 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c3939885-f5d4-4bd2-a69f-435c2a8fa048-etc-selinux\") pod \"aws-ebs-csi-driver-node-cq2wg\" (UID: \"c3939885-f5d4-4bd2-a69f-435c2a8fa048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:56.855848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855822 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855885 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f-metrics-certs\") pod \"network-metrics-daemon-pcvbh\" (UID: \"0fe8107c-9128-4c8a-9bde-e1cacd2ed19f\") " pod="openshift-multus/network-metrics-daemon-pcvbh" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855892 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c3939885-f5d4-4bd2-a69f-435c2a8fa048-registration-dir\") pod \"aws-ebs-csi-driver-node-cq2wg\" (UID: \"c3939885-f5d4-4bd2-a69f-435c2a8fa048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855914 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c3939885-f5d4-4bd2-a69f-435c2a8fa048-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cq2wg\" (UID: \"c3939885-f5d4-4bd2-a69f-435c2a8fa048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855917 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c3939885-f5d4-4bd2-a69f-435c2a8fa048-etc-selinux\") pod \"aws-ebs-csi-driver-node-cq2wg\" (UID: \"c3939885-f5d4-4bd2-a69f-435c2a8fa048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: E1031 00:55:56.855938 4440 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49f7c8f4-b979-436d-b035-20f9fcae140a-metrics-tls podName:49f7c8f4-b979-436d-b035-20f9fcae140a nodeName:}" failed. No retries permitted until 2026-10-31 00:55:57.355918503 +0000 UTC m=+1.714640828 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49f7c8f4-b979-436d-b035-20f9fcae140a-metrics-tls") pod "dns-default-zhmhj" (UID: "49f7c8f4-b979-436d-b035-20f9fcae140a") : secret "dns-default-metrics-tls" not found Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855952 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c3939885-f5d4-4bd2-a69f-435c2a8fa048-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cq2wg\" (UID: \"c3939885-f5d4-4bd2-a69f-435c2a8fa048\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq2wg" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855966 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-node-log\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855982 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-host-cni-netd\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: E1031 00:55:56.855968 4440 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.856019 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-host-cni-netd\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.856021 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-host-run-netns\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.856048 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-node-log\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: E1031 00:55:56.856082 4440 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f-metrics-certs podName:0fe8107c-9128-4c8a-9bde-e1cacd2ed19f nodeName:}" failed. No retries permitted until 2026-10-31 00:55:57.356070516 +0000 UTC m=+1.714792815 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f-metrics-certs") pod "network-metrics-daemon-pcvbh" (UID: "0fe8107c-9128-4c8a-9bde-e1cacd2ed19f") : secret "metrics-daemon-secret" not found Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.855997 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-host-run-netns\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.856108 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-sys\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.856125 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-system-cni-dir\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.856139 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-multus-conf-dir\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.856155 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b8ba0db-d8b9-4a40-a144-1400634cd99d-host-slash\") pod \"iptables-alerter-mtwcc\" (UID: \"4b8ba0db-d8b9-4a40-a144-1400634cd99d\") " pod="openshift-network-operator/iptables-alerter-mtwcc" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.856171 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-etc-modprobe-d\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.856185 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-sys\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.856196 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-multus-conf-dir\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.856212 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-etc-systemd\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.856216 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b8ba0db-d8b9-4a40-a144-1400634cd99d-host-slash\") pod \"iptables-alerter-mtwcc\" (UID: \"4b8ba0db-d8b9-4a40-a144-1400634cd99d\") " pod="openshift-network-operator/iptables-alerter-mtwcc" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.856234 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-system-cni-dir\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.856245 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-host-var-lib-cni-bin\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.856269 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-host-var-lib-kubelet\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.856286 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cd654994-98a0-4d39-aa38-270b4a0d4341-hosts-file\") pod \"node-resolver-qsppl\" (UID: \"cd654994-98a0-4d39-aa38-270b4a0d4341\") " pod="openshift-dns/node-resolver-qsppl" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.856288 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-host-var-lib-cni-bin\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.856293 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-host-var-lib-kubelet\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.856248 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-etc-systemd\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.856305 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-host-run-k8s-cni-cncf-io\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.856283 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-etc-modprobe-d\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.856319 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-var-lib-kubelet\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.856337 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/49f8a438-6b29-4160-98e1-3f26e27fe0af-kubelet-config\") pod \"global-pull-secret-syncer-s9tlh\" (UID: \"49f8a438-6b29-4160-98e1-3f26e27fe0af\") " pod="kube-system/global-pull-secret-syncer-s9tlh" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.856344 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-host-run-k8s-cni-cncf-io\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.856345 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cd654994-98a0-4d39-aa38-270b4a0d4341-hosts-file\") pod \"node-resolver-qsppl\" (UID: \"cd654994-98a0-4d39-aa38-270b4a0d4341\") " pod="openshift-dns/node-resolver-qsppl" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.856369 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88c1a03a-cb94-4226-9d53-cd923895569e-system-cni-dir\") pod \"multus-additional-cni-plugins-p8vd6\" (UID: \"88c1a03a-cb94-4226-9d53-cd923895569e\") " pod="openshift-multus/multus-additional-cni-plugins-p8vd6" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.856381 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6ddfc5f-887b-4821-bac9-54cb59c50e32-var-lib-kubelet\") pod \"tuned-8r7qf\" (UID: \"a6ddfc5f-887b-4821-bac9-54cb59c50e32\") " pod="openshift-cluster-node-tuning-operator/tuned-8r7qf" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.856390 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88c1a03a-cb94-4226-9d53-cd923895569e-system-cni-dir\") pod \"multus-additional-cni-plugins-p8vd6\" (UID: \"88c1a03a-cb94-4226-9d53-cd923895569e\") " pod="openshift-multus/multus-additional-cni-plugins-p8vd6" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.856402 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-host-kubelet\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.856384 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-host-kubelet\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.856450 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-run-systemd\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.856465 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-multus-cni-dir\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.856412 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/49f8a438-6b29-4160-98e1-3f26e27fe0af-kubelet-config\") pod \"global-pull-secret-syncer-s9tlh\" (UID: \"49f8a438-6b29-4160-98e1-3f26e27fe0af\") " pod="kube-system/global-pull-secret-syncer-s9tlh" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.856521 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d610021a-d4d9-473d-8889-cb5afcb5b8a0-multus-cni-dir\") pod \"multus-g5h4s\" (UID: \"d610021a-d4d9-473d-8889-cb5afcb5b8a0\") " pod="openshift-multus/multus-g5h4s" Oct 31 00:55:56.857361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:56.856530 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d-run-systemd\") pod \"ovnkube-node-vcdmb\" (UID: \"3af8f815-9e96-4a2c-b3ec-c0bf5a03b34d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:55:57.038178 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:57.038158 4440 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:55:57.137009 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:57.136973 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:57.246021 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:57.245991 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:57.359723 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:57.359656 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49f7c8f4-b979-436d-b035-20f9fcae140a-metrics-tls\") pod \"dns-default-zhmhj\" (UID: \"49f7c8f4-b979-436d-b035-20f9fcae140a\") " pod="openshift-dns/dns-default-zhmhj" Oct 31 00:55:57.359723 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:57.359691 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f-metrics-certs\") pod \"network-metrics-daemon-pcvbh\" (UID: \"0fe8107c-9128-4c8a-9bde-e1cacd2ed19f\") " pod="openshift-multus/network-metrics-daemon-pcvbh" Oct 31 00:55:57.359873 ip-10-0-130-39 kubenswrapper[4440]: E1031 00:55:57.359798 4440 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 00:55:57.359873 ip-10-0-130-39 kubenswrapper[4440]: E1031 00:55:57.359857 4440 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f-metrics-certs podName:0fe8107c-9128-4c8a-9bde-e1cacd2ed19f nodeName:}" failed. No retries permitted until 2026-10-31 00:55:58.359844452 +0000 UTC m=+2.718566746 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f-metrics-certs") pod "network-metrics-daemon-pcvbh" (UID: "0fe8107c-9128-4c8a-9bde-e1cacd2ed19f") : secret "metrics-daemon-secret" not found Oct 31 00:55:57.359952 ip-10-0-130-39 kubenswrapper[4440]: E1031 00:55:57.359801 4440 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 00:55:57.359986 ip-10-0-130-39 kubenswrapper[4440]: E1031 00:55:57.359951 4440 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49f7c8f4-b979-436d-b035-20f9fcae140a-metrics-tls podName:49f7c8f4-b979-436d-b035-20f9fcae140a nodeName:}" failed. No retries permitted until 2026-10-31 00:55:58.359937143 +0000 UTC m=+2.718659440 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49f7c8f4-b979-436d-b035-20f9fcae140a-metrics-tls") pod "dns-default-zhmhj" (UID: "49f7c8f4-b979-436d-b035-20f9fcae140a") : secret "dns-default-metrics-tls" not found Oct 31 00:55:57.804693 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:57.804650 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-sk9vt" event={"ID":"ee8bc72c-f181-452d-ab71-b13a51672bcf","Type":"ContainerStarted","Data":"63456be3e2f90b5067b0c1359ca6252f76fc863d4be8985ccc6ee65cfb69a8a0"} Oct 31 00:55:58.367817 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:58.367777 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49f7c8f4-b979-436d-b035-20f9fcae140a-metrics-tls\") pod \"dns-default-zhmhj\" (UID: \"49f7c8f4-b979-436d-b035-20f9fcae140a\") " pod="openshift-dns/dns-default-zhmhj" Oct 31 00:55:58.367817 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:58.367824 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f-metrics-certs\") pod \"network-metrics-daemon-pcvbh\" (UID: \"0fe8107c-9128-4c8a-9bde-e1cacd2ed19f\") " pod="openshift-multus/network-metrics-daemon-pcvbh" Oct 31 00:55:58.368091 ip-10-0-130-39 kubenswrapper[4440]: E1031 00:55:58.367944 4440 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 00:55:58.368091 ip-10-0-130-39 kubenswrapper[4440]: E1031 00:55:58.368007 4440 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f-metrics-certs podName:0fe8107c-9128-4c8a-9bde-e1cacd2ed19f nodeName:}" failed. No retries permitted until 2026-10-31 00:56:00.367994284 +0000 UTC m=+4.726716591 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f-metrics-certs") pod "network-metrics-daemon-pcvbh" (UID: "0fe8107c-9128-4c8a-9bde-e1cacd2ed19f") : secret "metrics-daemon-secret" not found Oct 31 00:55:58.368091 ip-10-0-130-39 kubenswrapper[4440]: E1031 00:55:58.367944 4440 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 00:55:58.368091 ip-10-0-130-39 kubenswrapper[4440]: E1031 00:55:58.368082 4440 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49f7c8f4-b979-436d-b035-20f9fcae140a-metrics-tls podName:49f7c8f4-b979-436d-b035-20f9fcae140a nodeName:}" failed. No retries permitted until 2026-10-31 00:56:00.36806965 +0000 UTC m=+4.726791959 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49f7c8f4-b979-436d-b035-20f9fcae140a-metrics-tls") pod "dns-default-zhmhj" (UID: "49f7c8f4-b979-436d-b035-20f9fcae140a") : secret "dns-default-metrics-tls" not found Oct 31 00:55:58.408364 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:58.408323 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:58.693772 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:58.693731 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:58.817939 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:58.817907 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:59.305539 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:59.305507 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:59.665438 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:59.665341 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:59.793215 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:55:59.793179 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:00.093726 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:00.093690 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:00.299400 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:00.299369 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:00.382927 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:00.382855 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49f7c8f4-b979-436d-b035-20f9fcae140a-metrics-tls\") pod \"dns-default-zhmhj\" (UID: \"49f7c8f4-b979-436d-b035-20f9fcae140a\") " pod="openshift-dns/dns-default-zhmhj" Oct 31 00:56:00.382927 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:00.382895 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f-metrics-certs\") pod \"network-metrics-daemon-pcvbh\" (UID: \"0fe8107c-9128-4c8a-9bde-e1cacd2ed19f\") " pod="openshift-multus/network-metrics-daemon-pcvbh" Oct 31 00:56:00.383090 ip-10-0-130-39 kubenswrapper[4440]: E1031 00:56:00.383005 4440 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 00:56:00.383090 ip-10-0-130-39 kubenswrapper[4440]: E1031 00:56:00.383074 4440 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49f7c8f4-b979-436d-b035-20f9fcae140a-metrics-tls podName:49f7c8f4-b979-436d-b035-20f9fcae140a nodeName:}" failed. No retries permitted until 2026-10-31 00:56:04.383059551 +0000 UTC m=+8.741781845 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49f7c8f4-b979-436d-b035-20f9fcae140a-metrics-tls") pod "dns-default-zhmhj" (UID: "49f7c8f4-b979-436d-b035-20f9fcae140a") : secret "dns-default-metrics-tls" not found Oct 31 00:56:00.383266 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:00.383247 4440 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 00:56:00.385375 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:00.385349 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fe8107c-9128-4c8a-9bde-e1cacd2ed19f-metrics-certs\") pod \"network-metrics-daemon-pcvbh\" (UID: \"0fe8107c-9128-4c8a-9bde-e1cacd2ed19f\") " pod="openshift-multus/network-metrics-daemon-pcvbh" Oct 31 00:56:00.423164 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:00.423134 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:00.536621 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:00.536579 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:00.633284 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:00.633194 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pcvbh" Oct 31 00:56:00.688099 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:00.688063 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:00.916115 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:00.916025 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pcvbh"] Oct 31 00:56:00.920301 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:56:00.920270 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fe8107c_9128_4c8a_9bde_e1cacd2ed19f.slice/crio-e5b3ca519220ffeca36b58bf46c7fb1a777125329e7ebd62f8d475b01fe58d2b WatchSource:0}: Error finding container e5b3ca519220ffeca36b58bf46c7fb1a777125329e7ebd62f8d475b01fe58d2b: Status 404 returned error can't find the container with id e5b3ca519220ffeca36b58bf46c7fb1a777125329e7ebd62f8d475b01fe58d2b Oct 31 00:56:01.540803 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:01.540767 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:01.670851 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:01.670806 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:01.816180 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:01.816096 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pcvbh" event={"ID":"0fe8107c-9128-4c8a-9bde-e1cacd2ed19f","Type":"ContainerStarted","Data":"e5b3ca519220ffeca36b58bf46c7fb1a777125329e7ebd62f8d475b01fe58d2b"} Oct 31 00:56:01.837750 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:01.837712 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:01.978004 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:01.977972 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:02.090497 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:02.090385 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:02.269721 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:02.269685 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:02.320905 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:02.320867 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5xvng" Oct 31 00:56:02.480913 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:02.480877 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:02.522296 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:02.522264 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:56:02.686464 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:02.686412 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:02.820141 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:02.820051 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pcvbh" event={"ID":"0fe8107c-9128-4c8a-9bde-e1cacd2ed19f","Type":"ContainerStarted","Data":"4601739f41520fe7e9a893f04f43b8f780bae2f86c1f2bda9e34dbcb8408dc0f"} Oct 31 00:56:02.820141 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:02.820103 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pcvbh" event={"ID":"0fe8107c-9128-4c8a-9bde-e1cacd2ed19f","Type":"ContainerStarted","Data":"5414b0e21525441a92c7b6bbcaaf1d194d726c9e9008da8db465d57bd2f62f62"} Oct 31 00:56:03.531104 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:03.531071 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:03.679862 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:03.679826 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:03.789593 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:03.789502 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:03.913109 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:03.913075 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:04.074629 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:04.074538 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:04.270684 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:04.270653 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:04.410230 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:04.410144 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49f7c8f4-b979-436d-b035-20f9fcae140a-metrics-tls\") pod \"dns-default-zhmhj\" (UID: \"49f7c8f4-b979-436d-b035-20f9fcae140a\") " pod="openshift-dns/dns-default-zhmhj" Oct 31 00:56:04.412588 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:04.412568 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49f7c8f4-b979-436d-b035-20f9fcae140a-metrics-tls\") pod \"dns-default-zhmhj\" (UID: \"49f7c8f4-b979-436d-b035-20f9fcae140a\") " pod="openshift-dns/dns-default-zhmhj" Oct 31 00:56:04.478078 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:04.478049 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:04.535749 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:04.535716 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zhmhj" Oct 31 00:56:04.738671 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:04.738643 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:04.808590 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:04.808553 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zhmhj"] Oct 31 00:56:04.811994 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:56:04.811957 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49f7c8f4_b979_436d_b035_20f9fcae140a.slice/crio-c2bf4f80282a2afdf1ae048d384227b743bdf86afae1fe104618ef61811998e7 WatchSource:0}: Error finding container c2bf4f80282a2afdf1ae048d384227b743bdf86afae1fe104618ef61811998e7: Status 404 returned error can't find the container with id c2bf4f80282a2afdf1ae048d384227b743bdf86afae1fe104618ef61811998e7 Oct 31 00:56:04.827950 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:04.827925 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zhmhj" event={"ID":"49f7c8f4-b979-436d-b035-20f9fcae140a","Type":"ContainerStarted","Data":"c2bf4f80282a2afdf1ae048d384227b743bdf86afae1fe104618ef61811998e7"} Oct 31 00:56:05.565160 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:05.565119 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:05.671583 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:05.671546 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:05.872668 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:05.872591 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:06.068759 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:06.068725 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:06.126960 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:06.126885 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:56:06.127124 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:06.127114 4440 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 00:56:06.264897 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:06.264859 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:06.462976 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:06.462939 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:06.674116 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:06.674077 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:07.840069 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:07.840029 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zhmhj" event={"ID":"49f7c8f4-b979-436d-b035-20f9fcae140a","Type":"ContainerStarted","Data":"469c9c9586a7380371bfa7f34cf454996179a35cf6fdcacac6a41690c415c21b"} Oct 31 00:56:07.840480 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:07.840081 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zhmhj" event={"ID":"49f7c8f4-b979-436d-b035-20f9fcae140a","Type":"ContainerStarted","Data":"957f0b240a09dff988d98332dcc162be25da4778d8e3808b7047049fc0999872"} Oct 31 00:56:07.840480 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:07.840188 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zhmhj" Oct 31 00:56:07.881884 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:07.881850 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:08.066872 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:08.066833 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:08.264982 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:08.264949 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:08.463361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:08.463327 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:08.679337 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:08.679303 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:09.066213 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:09.066138 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:09.169843 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:09.169779 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:09.464471 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:09.464437 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:13.493754 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:13.493713 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:13.721988 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:13.721960 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8kln5" Oct 31 00:56:15.228225 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.228191 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lglrb"] Oct 31 00:56:15.230389 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.230373 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lglrb" Oct 31 00:56:15.254550 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.254507 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 00:56:15.264379 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.264349 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lglrb"] Oct 31 00:56:15.269696 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.269673 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-gjjhv"] Oct 31 00:56:15.271505 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.271487 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-gjjhv" Oct 31 00:56:15.283396 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.283377 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 00:56:15.283520 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.283506 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 00:56:15.283863 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.283851 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 31 00:56:15.283908 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.283878 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a2b4a048-eaac-436d-8439-61a585d03601-data-volume\") pod \"insights-runtime-extractor-lglrb\" (UID: \"a2b4a048-eaac-436d-8439-61a585d03601\") " pod="openshift-insights/insights-runtime-extractor-lglrb" Oct 31 00:56:15.283949 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.283910 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsndd\" (UniqueName: \"kubernetes.io/projected/a2b4a048-eaac-436d-8439-61a585d03601-kube-api-access-xsndd\") pod \"insights-runtime-extractor-lglrb\" (UID: \"a2b4a048-eaac-436d-8439-61a585d03601\") " pod="openshift-insights/insights-runtime-extractor-lglrb" Oct 31 00:56:15.283949 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.283937 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a2b4a048-eaac-436d-8439-61a585d03601-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lglrb\" (UID: \"a2b4a048-eaac-436d-8439-61a585d03601\") " pod="openshift-insights/insights-runtime-extractor-lglrb" Oct 31 00:56:15.284015 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.283982 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a2b4a048-eaac-436d-8439-61a585d03601-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lglrb\" (UID: \"a2b4a048-eaac-436d-8439-61a585d03601\") " pod="openshift-insights/insights-runtime-extractor-lglrb" Oct 31 00:56:15.284119 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.284103 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a2b4a048-eaac-436d-8439-61a585d03601-crio-socket\") pod \"insights-runtime-extractor-lglrb\" (UID: \"a2b4a048-eaac-436d-8439-61a585d03601\") " pod="openshift-insights/insights-runtime-extractor-lglrb" Oct 31 00:56:15.290907 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.290885 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-gjjhv"] Oct 31 00:56:15.303377 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.303355 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 31 00:56:15.341358 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.341321 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p5vm9\"" Oct 31 00:56:15.341566 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.341413 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 00:56:15.341566 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.341517 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-nw48m\"" Oct 31 00:56:15.343806 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.343784 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-85c74bc6cd-ggkcd"] Oct 31 00:56:15.345281 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.345259 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:15.345646 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.345628 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85c74bc6cd-ggkcd" Oct 31 00:56:15.359567 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.359543 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 31 00:56:15.365139 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.365117 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85c74bc6cd-ggkcd"] Oct 31 00:56:15.385168 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.385138 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ed6c333-706a-44dc-aae6-54be82855800-tmp\") pod \"downloads-6dd789c8b7-gjjhv\" (UID: \"6ed6c333-706a-44dc-aae6-54be82855800\") " pod="openshift-console/downloads-6dd789c8b7-gjjhv" Oct 31 00:56:15.385286 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.385178 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a2b4a048-eaac-436d-8439-61a585d03601-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lglrb\" (UID: \"a2b4a048-eaac-436d-8439-61a585d03601\") " pod="openshift-insights/insights-runtime-extractor-lglrb" Oct 31 00:56:15.385286 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.385201 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk47c\" (UniqueName: \"kubernetes.io/projected/faccb6fb-036d-4190-a0e8-88f417c6a29b-kube-api-access-wk47c\") pod \"image-registry-85c74bc6cd-ggkcd\" (UID: \"faccb6fb-036d-4190-a0e8-88f417c6a29b\") " pod="openshift-image-registry/image-registry-85c74bc6cd-ggkcd" Oct 31 00:56:15.385286 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.385246 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a2b4a048-eaac-436d-8439-61a585d03601-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lglrb\" (UID: \"a2b4a048-eaac-436d-8439-61a585d03601\") " pod="openshift-insights/insights-runtime-extractor-lglrb" Oct 31 00:56:15.385436 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.385287 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4fvc\" (UniqueName: \"kubernetes.io/projected/6ed6c333-706a-44dc-aae6-54be82855800-kube-api-access-g4fvc\") pod \"downloads-6dd789c8b7-gjjhv\" (UID: \"6ed6c333-706a-44dc-aae6-54be82855800\") " pod="openshift-console/downloads-6dd789c8b7-gjjhv" Oct 31 00:56:15.385436 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.385331 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/faccb6fb-036d-4190-a0e8-88f417c6a29b-ca-trust-extracted\") pod \"image-registry-85c74bc6cd-ggkcd\" (UID: \"faccb6fb-036d-4190-a0e8-88f417c6a29b\") " pod="openshift-image-registry/image-registry-85c74bc6cd-ggkcd" Oct 31 00:56:15.385436 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.385351 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/faccb6fb-036d-4190-a0e8-88f417c6a29b-installation-pull-secrets\") pod \"image-registry-85c74bc6cd-ggkcd\" (UID: \"faccb6fb-036d-4190-a0e8-88f417c6a29b\") " pod="openshift-image-registry/image-registry-85c74bc6cd-ggkcd" Oct 31 00:56:15.385599 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.385442 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a2b4a048-eaac-436d-8439-61a585d03601-crio-socket\") pod \"insights-runtime-extractor-lglrb\" (UID: \"a2b4a048-eaac-436d-8439-61a585d03601\") " pod="openshift-insights/insights-runtime-extractor-lglrb" Oct 31 00:56:15.385599 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.385480 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/faccb6fb-036d-4190-a0e8-88f417c6a29b-image-registry-private-configuration\") pod \"image-registry-85c74bc6cd-ggkcd\" (UID: \"faccb6fb-036d-4190-a0e8-88f417c6a29b\") " pod="openshift-image-registry/image-registry-85c74bc6cd-ggkcd" Oct 31 00:56:15.385599 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.385500 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/faccb6fb-036d-4190-a0e8-88f417c6a29b-registry-certificates\") pod \"image-registry-85c74bc6cd-ggkcd\" (UID: \"faccb6fb-036d-4190-a0e8-88f417c6a29b\") " pod="openshift-image-registry/image-registry-85c74bc6cd-ggkcd" Oct 31 00:56:15.385599 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.385527 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/faccb6fb-036d-4190-a0e8-88f417c6a29b-trusted-ca\") pod \"image-registry-85c74bc6cd-ggkcd\" (UID: \"faccb6fb-036d-4190-a0e8-88f417c6a29b\") " pod="openshift-image-registry/image-registry-85c74bc6cd-ggkcd" Oct 31 00:56:15.385599 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.385559 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a2b4a048-eaac-436d-8439-61a585d03601-crio-socket\") pod \"insights-runtime-extractor-lglrb\" (UID: \"a2b4a048-eaac-436d-8439-61a585d03601\") " pod="openshift-insights/insights-runtime-extractor-lglrb" Oct 31 00:56:15.385599 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.385588 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/faccb6fb-036d-4190-a0e8-88f417c6a29b-bound-sa-token\") pod \"image-registry-85c74bc6cd-ggkcd\" (UID: \"faccb6fb-036d-4190-a0e8-88f417c6a29b\") " pod="openshift-image-registry/image-registry-85c74bc6cd-ggkcd" Oct 31 00:56:15.385906 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.385624 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a2b4a048-eaac-436d-8439-61a585d03601-data-volume\") pod \"insights-runtime-extractor-lglrb\" (UID: \"a2b4a048-eaac-436d-8439-61a585d03601\") " pod="openshift-insights/insights-runtime-extractor-lglrb" Oct 31 00:56:15.385906 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.385702 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsndd\" (UniqueName: \"kubernetes.io/projected/a2b4a048-eaac-436d-8439-61a585d03601-kube-api-access-xsndd\") pod \"insights-runtime-extractor-lglrb\" (UID: \"a2b4a048-eaac-436d-8439-61a585d03601\") " pod="openshift-insights/insights-runtime-extractor-lglrb" Oct 31 00:56:15.385906 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.385742 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/faccb6fb-036d-4190-a0e8-88f417c6a29b-registry-tls\") pod \"image-registry-85c74bc6cd-ggkcd\" (UID: \"faccb6fb-036d-4190-a0e8-88f417c6a29b\") " pod="openshift-image-registry/image-registry-85c74bc6cd-ggkcd" Oct 31 00:56:15.385906 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.385869 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a2b4a048-eaac-436d-8439-61a585d03601-data-volume\") pod \"insights-runtime-extractor-lglrb\" (UID: \"a2b4a048-eaac-436d-8439-61a585d03601\") " pod="openshift-insights/insights-runtime-extractor-lglrb" Oct 31 00:56:15.386090 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.386013 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a2b4a048-eaac-436d-8439-61a585d03601-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lglrb\" (UID: \"a2b4a048-eaac-436d-8439-61a585d03601\") " pod="openshift-insights/insights-runtime-extractor-lglrb" Oct 31 00:56:15.387649 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.387633 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a2b4a048-eaac-436d-8439-61a585d03601-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lglrb\" (UID: \"a2b4a048-eaac-436d-8439-61a585d03601\") " pod="openshift-insights/insights-runtime-extractor-lglrb" Oct 31 00:56:15.397607 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.397579 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsndd\" (UniqueName: \"kubernetes.io/projected/a2b4a048-eaac-436d-8439-61a585d03601-kube-api-access-xsndd\") pod \"insights-runtime-extractor-lglrb\" (UID: \"a2b4a048-eaac-436d-8439-61a585d03601\") " pod="openshift-insights/insights-runtime-extractor-lglrb" Oct 31 00:56:15.429793 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.429758 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 31 00:56:15.429968 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.429856 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xgrpm\"" Oct 31 00:56:15.429968 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.429880 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 31 00:56:15.430078 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.430006 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 31 00:56:15.487066 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.486977 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/faccb6fb-036d-4190-a0e8-88f417c6a29b-image-registry-private-configuration\") pod \"image-registry-85c74bc6cd-ggkcd\" (UID: \"faccb6fb-036d-4190-a0e8-88f417c6a29b\") " pod="openshift-image-registry/image-registry-85c74bc6cd-ggkcd" Oct 31 00:56:15.487066 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.487011 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/faccb6fb-036d-4190-a0e8-88f417c6a29b-registry-certificates\") pod \"image-registry-85c74bc6cd-ggkcd\" (UID: \"faccb6fb-036d-4190-a0e8-88f417c6a29b\") " pod="openshift-image-registry/image-registry-85c74bc6cd-ggkcd" Oct 31 00:56:15.487066 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.487030 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/faccb6fb-036d-4190-a0e8-88f417c6a29b-trusted-ca\") pod \"image-registry-85c74bc6cd-ggkcd\" (UID: \"faccb6fb-036d-4190-a0e8-88f417c6a29b\") " pod="openshift-image-registry/image-registry-85c74bc6cd-ggkcd" Oct 31 00:56:15.487066 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.487063 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/faccb6fb-036d-4190-a0e8-88f417c6a29b-bound-sa-token\") pod \"image-registry-85c74bc6cd-ggkcd\" (UID: \"faccb6fb-036d-4190-a0e8-88f417c6a29b\") " pod="openshift-image-registry/image-registry-85c74bc6cd-ggkcd" Oct 31 00:56:15.487355 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.487105 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/faccb6fb-036d-4190-a0e8-88f417c6a29b-registry-tls\") pod \"image-registry-85c74bc6cd-ggkcd\" (UID: \"faccb6fb-036d-4190-a0e8-88f417c6a29b\") " pod="openshift-image-registry/image-registry-85c74bc6cd-ggkcd" Oct 31 00:56:15.487355 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.487138 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ed6c333-706a-44dc-aae6-54be82855800-tmp\") pod \"downloads-6dd789c8b7-gjjhv\" (UID: \"6ed6c333-706a-44dc-aae6-54be82855800\") " pod="openshift-console/downloads-6dd789c8b7-gjjhv" Oct 31 00:56:15.487355 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.487167 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk47c\" (UniqueName: \"kubernetes.io/projected/faccb6fb-036d-4190-a0e8-88f417c6a29b-kube-api-access-wk47c\") pod \"image-registry-85c74bc6cd-ggkcd\" (UID: \"faccb6fb-036d-4190-a0e8-88f417c6a29b\") " pod="openshift-image-registry/image-registry-85c74bc6cd-ggkcd" Oct 31 00:56:15.487355 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.487283 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4fvc\" (UniqueName: \"kubernetes.io/projected/6ed6c333-706a-44dc-aae6-54be82855800-kube-api-access-g4fvc\") pod \"downloads-6dd789c8b7-gjjhv\" (UID: \"6ed6c333-706a-44dc-aae6-54be82855800\") " pod="openshift-console/downloads-6dd789c8b7-gjjhv" Oct 31 00:56:15.487355 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.487326 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/faccb6fb-036d-4190-a0e8-88f417c6a29b-ca-trust-extracted\") pod \"image-registry-85c74bc6cd-ggkcd\" (UID: \"faccb6fb-036d-4190-a0e8-88f417c6a29b\") " pod="openshift-image-registry/image-registry-85c74bc6cd-ggkcd" Oct 31 00:56:15.487355 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.487350 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/faccb6fb-036d-4190-a0e8-88f417c6a29b-installation-pull-secrets\") pod \"image-registry-85c74bc6cd-ggkcd\" (UID: \"faccb6fb-036d-4190-a0e8-88f417c6a29b\") " pod="openshift-image-registry/image-registry-85c74bc6cd-ggkcd" Oct 31 00:56:15.487809 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.487780 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/faccb6fb-036d-4190-a0e8-88f417c6a29b-ca-trust-extracted\") pod \"image-registry-85c74bc6cd-ggkcd\" (UID: \"faccb6fb-036d-4190-a0e8-88f417c6a29b\") " pod="openshift-image-registry/image-registry-85c74bc6cd-ggkcd" Oct 31 00:56:15.487994 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.487974 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/faccb6fb-036d-4190-a0e8-88f417c6a29b-registry-certificates\") pod \"image-registry-85c74bc6cd-ggkcd\" (UID: \"faccb6fb-036d-4190-a0e8-88f417c6a29b\") " pod="openshift-image-registry/image-registry-85c74bc6cd-ggkcd" Oct 31 00:56:15.488083 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.487995 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ed6c333-706a-44dc-aae6-54be82855800-tmp\") pod \"downloads-6dd789c8b7-gjjhv\" (UID: \"6ed6c333-706a-44dc-aae6-54be82855800\") " pod="openshift-console/downloads-6dd789c8b7-gjjhv" Oct 31 00:56:15.488146 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.488076 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/faccb6fb-036d-4190-a0e8-88f417c6a29b-trusted-ca\") pod \"image-registry-85c74bc6cd-ggkcd\" (UID: \"faccb6fb-036d-4190-a0e8-88f417c6a29b\") " pod="openshift-image-registry/image-registry-85c74bc6cd-ggkcd" Oct 31 00:56:15.489725 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.489706 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/faccb6fb-036d-4190-a0e8-88f417c6a29b-registry-tls\") pod \"image-registry-85c74bc6cd-ggkcd\" (UID: \"faccb6fb-036d-4190-a0e8-88f417c6a29b\") " pod="openshift-image-registry/image-registry-85c74bc6cd-ggkcd" Oct 31 00:56:15.489987 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.489970 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/faccb6fb-036d-4190-a0e8-88f417c6a29b-image-registry-private-configuration\") pod \"image-registry-85c74bc6cd-ggkcd\" (UID: \"faccb6fb-036d-4190-a0e8-88f417c6a29b\") " pod="openshift-image-registry/image-registry-85c74bc6cd-ggkcd" Oct 31 00:56:15.490137 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.490118 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/faccb6fb-036d-4190-a0e8-88f417c6a29b-installation-pull-secrets\") pod \"image-registry-85c74bc6cd-ggkcd\" (UID: \"faccb6fb-036d-4190-a0e8-88f417c6a29b\") " pod="openshift-image-registry/image-registry-85c74bc6cd-ggkcd" Oct 31 00:56:15.498130 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.498098 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/faccb6fb-036d-4190-a0e8-88f417c6a29b-bound-sa-token\") pod \"image-registry-85c74bc6cd-ggkcd\" (UID: \"faccb6fb-036d-4190-a0e8-88f417c6a29b\") " pod="openshift-image-registry/image-registry-85c74bc6cd-ggkcd" Oct 31 00:56:15.498316 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.498297 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk47c\" (UniqueName: \"kubernetes.io/projected/faccb6fb-036d-4190-a0e8-88f417c6a29b-kube-api-access-wk47c\") pod \"image-registry-85c74bc6cd-ggkcd\" (UID: \"faccb6fb-036d-4190-a0e8-88f417c6a29b\") " pod="openshift-image-registry/image-registry-85c74bc6cd-ggkcd" Oct 31 00:56:15.498646 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.498630 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4fvc\" (UniqueName: \"kubernetes.io/projected/6ed6c333-706a-44dc-aae6-54be82855800-kube-api-access-g4fvc\") pod \"downloads-6dd789c8b7-gjjhv\" (UID: \"6ed6c333-706a-44dc-aae6-54be82855800\") " pod="openshift-console/downloads-6dd789c8b7-gjjhv" Oct 31 00:56:15.541128 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.541095 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:15.542941 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.542927 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lglrb" Oct 31 00:56:15.581761 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.581738 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-gjjhv" Oct 31 00:56:15.657347 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.657324 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85c74bc6cd-ggkcd" Oct 31 00:56:15.679236 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.679213 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:15.789217 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.789129 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:15.813333 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.813302 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lglrb"] Oct 31 00:56:15.817128 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:56:15.817098 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2b4a048_eaac_436d_8439_61a585d03601.slice/crio-014f493965b67c102e7457031069c3fdc76a99fab86be7dba91e6e8de5e77b52 WatchSource:0}: Error finding container 014f493965b67c102e7457031069c3fdc76a99fab86be7dba91e6e8de5e77b52: Status 404 returned error can't find the container with id 014f493965b67c102e7457031069c3fdc76a99fab86be7dba91e6e8de5e77b52 Oct 31 00:56:15.857255 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.857231 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-gjjhv"] Oct 31 00:56:15.860084 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:56:15.860056 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed6c333_706a_44dc_aae6_54be82855800.slice/crio-52ac28757c3e9313793bbbe3fd09e8b39bfd2b9269277e54e193797cc837aceb WatchSource:0}: Error finding container 52ac28757c3e9313793bbbe3fd09e8b39bfd2b9269277e54e193797cc837aceb: Status 404 returned error can't find the container with id 52ac28757c3e9313793bbbe3fd09e8b39bfd2b9269277e54e193797cc837aceb Oct 31 00:56:15.864998 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.864976 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lglrb" event={"ID":"a2b4a048-eaac-436d-8439-61a585d03601","Type":"ContainerStarted","Data":"25eba9a4d182e29640f5a7680401148d7b59d07df4970c587fcd662bff6fab9c"} Oct 31 00:56:15.865085 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.865022 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lglrb" event={"ID":"a2b4a048-eaac-436d-8439-61a585d03601","Type":"ContainerStarted","Data":"014f493965b67c102e7457031069c3fdc76a99fab86be7dba91e6e8de5e77b52"} Oct 31 00:56:15.866086 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.866066 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-gjjhv" event={"ID":"6ed6c333-706a-44dc-aae6-54be82855800","Type":"ContainerStarted","Data":"52ac28757c3e9313793bbbe3fd09e8b39bfd2b9269277e54e193797cc837aceb"} Oct 31 00:56:15.928312 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:15.927896 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85c74bc6cd-ggkcd"] Oct 31 00:56:15.932311 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:56:15.932281 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaccb6fb_036d_4190_a0e8_88f417c6a29b.slice/crio-03a1ffff3805d4e1f80f0357e8338479e9e443b23c49499e74b6b05cdeedb0aa WatchSource:0}: Error finding container 03a1ffff3805d4e1f80f0357e8338479e9e443b23c49499e74b6b05cdeedb0aa: Status 404 returned error can't find the container with id 03a1ffff3805d4e1f80f0357e8338479e9e443b23c49499e74b6b05cdeedb0aa Oct 31 00:56:16.872412 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:16.872377 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lglrb" event={"ID":"a2b4a048-eaac-436d-8439-61a585d03601","Type":"ContainerStarted","Data":"e2d3facee01edb84e9afd2cfe0a8a4c4c9ca7513660fbfae31dc093326658cce"} Oct 31 00:56:16.874170 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:16.874140 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85c74bc6cd-ggkcd" event={"ID":"faccb6fb-036d-4190-a0e8-88f417c6a29b","Type":"ContainerStarted","Data":"6534badc2c7913025bd51e3891cc36d0f13c909220cd5d0589691d6bc3e90895"} Oct 31 00:56:16.874294 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:16.874192 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85c74bc6cd-ggkcd" event={"ID":"faccb6fb-036d-4190-a0e8-88f417c6a29b","Type":"ContainerStarted","Data":"03a1ffff3805d4e1f80f0357e8338479e9e443b23c49499e74b6b05cdeedb0aa"} Oct 31 00:56:16.874294 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:16.874245 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-85c74bc6cd-ggkcd" Oct 31 00:56:17.246651 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:17.246587 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:17.357972 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:17.357891 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:17.845576 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:17.845541 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zhmhj" Oct 31 00:56:17.864720 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:17.864653 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-85c74bc6cd-ggkcd" podStartSLOduration=2.864634101 podStartE2EDuration="2.864634101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:16.894645937 +0000 UTC m=+21.253368355" watchObservedRunningTime="2026-10-31 00:56:17.864634101 +0000 UTC m=+22.223356418" Oct 31 00:56:18.009935 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:18.009845 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:18.089358 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:18.089325 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zhmhj_49f7c8f4-b979-436d-b035-20f9fcae140a/dns/0.log" Oct 31 00:56:18.089565 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:18.089390 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zhmhj_49f7c8f4-b979-436d-b035-20f9fcae140a/dns/0.log" Oct 31 00:56:18.125625 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:18.125536 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:18.288468 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:18.288441 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zhmhj_49f7c8f4-b979-436d-b035-20f9fcae140a/kube-rbac-proxy/0.log" Oct 31 00:56:18.288835 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:18.288820 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zhmhj_49f7c8f4-b979-436d-b035-20f9fcae140a/kube-rbac-proxy/0.log" Oct 31 00:56:18.396660 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:18.396580 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:18.507125 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:18.507095 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:18.693741 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:18.693710 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qsppl_cd654994-98a0-4d39-aa38-270b4a0d4341/dns-node-resolver/0.log" Oct 31 00:56:18.693915 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:18.693715 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qsppl_cd654994-98a0-4d39-aa38-270b4a0d4341/dns-node-resolver/0.log" Oct 31 00:56:18.863518 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:18.863488 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:18.885292 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:18.885257 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lglrb" event={"ID":"a2b4a048-eaac-436d-8439-61a585d03601","Type":"ContainerStarted","Data":"c03d1e892040b47b68f4c91f8a58d948b87e93b4b06156cde650fd68a88723ac"} Oct 31 00:56:18.952223 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:18.952034 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-lglrb" podStartSLOduration=1.6458324530000001 podStartE2EDuration="3.952017058s" totalImagesPullingTime="2.306184605s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:56:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:18.951948182 +0000 UTC m=+23.310670497" watchObservedRunningTime="2026-10-31 00:56:18.952017058 +0000 UTC m=+23.310739375" Oct 31 00:56:19.013787 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:19.013744 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:19.290175 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:19.290060 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-85c74bc6cd-ggkcd_faccb6fb-036d-4190-a0e8-88f417c6a29b/registry/0.log" Oct 31 00:56:19.290368 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:19.290355 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-85c74bc6cd-ggkcd_faccb6fb-036d-4190-a0e8-88f417c6a29b/registry/0.log" Oct 31 00:56:19.677178 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:19.677143 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:19.812046 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:19.812000 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:19.888650 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:19.888615 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qzm2t_0a37f05b-0755-4ca7-8d9b-12030234bec1/node-ca/0.log" Oct 31 00:56:19.888853 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:19.888816 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qzm2t_0a37f05b-0755-4ca7-8d9b-12030234bec1/node-ca/0.log" Oct 31 00:56:19.935201 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:19.935118 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:20.064794 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:20.064756 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:20.255997 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:20.255905 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:20.491811 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:20.491771 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-srw8h_9a11b541-6b88-439e-bcbe-5901cb4c20b0/serve-healthcheck-canary/0.log" Oct 31 00:56:20.491811 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:20.491789 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-srw8h_9a11b541-6b88-439e-bcbe-5901cb4c20b0/serve-healthcheck-canary/0.log" Oct 31 00:56:20.810152 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:20.810115 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:21.107312 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:21.107232 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vcdmb" Oct 31 00:56:21.230983 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:21.230947 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:21.326682 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:21.326646 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:22.193472 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:22.193435 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:22.306872 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:22.306836 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:22.601400 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:22.601317 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:22.706386 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:22.706351 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:23.118816 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:23.118787 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:23.254849 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:23.254813 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:24.500283 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:24.500242 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:24.621763 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:24.621729 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:24.877501 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:24.877403 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:24.989717 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:24.989675 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:25.255067 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:25.255029 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:25.382684 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:25.382647 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:25.543325 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:25.543239 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:25.699934 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:25.699827 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:27.265180 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.265144 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:27.401153 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.401115 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:27.510037 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.510000 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6qp5m"] Oct 31 00:56:27.542763 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.542687 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6qp5m" Oct 31 00:56:27.565635 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.565589 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 00:56:27.566523 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.566499 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 00:56:27.566523 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.566509 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qgqml\"" Oct 31 00:56:27.571009 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.570984 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:27.606674 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.606644 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:56:27.606897 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.606873 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 00:56:27.606988 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.606924 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:56:27.606988 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.606953 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 00:56:27.689309 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.689278 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dd841d54-1f08-42b8-ac54-4d1aac4f81e8-node-exporter-textfile\") pod \"node-exporter-6qp5m\" (UID: \"dd841d54-1f08-42b8-ac54-4d1aac4f81e8\") " pod="openshift-monitoring/node-exporter-6qp5m" Oct 31 00:56:27.689509 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.689326 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dd841d54-1f08-42b8-ac54-4d1aac4f81e8-node-exporter-wtmp\") pod \"node-exporter-6qp5m\" (UID: \"dd841d54-1f08-42b8-ac54-4d1aac4f81e8\") " pod="openshift-monitoring/node-exporter-6qp5m" Oct 31 00:56:27.689509 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.689347 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd841d54-1f08-42b8-ac54-4d1aac4f81e8-sys\") pod \"node-exporter-6qp5m\" (UID: \"dd841d54-1f08-42b8-ac54-4d1aac4f81e8\") " pod="openshift-monitoring/node-exporter-6qp5m" Oct 31 00:56:27.689509 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.689365 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dd841d54-1f08-42b8-ac54-4d1aac4f81e8-node-exporter-accelerators-collector-config\") pod \"node-exporter-6qp5m\" (UID: \"dd841d54-1f08-42b8-ac54-4d1aac4f81e8\") " pod="openshift-monitoring/node-exporter-6qp5m" Oct 31 00:56:27.689509 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.689384 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd841d54-1f08-42b8-ac54-4d1aac4f81e8-metrics-client-ca\") pod \"node-exporter-6qp5m\" (UID: \"dd841d54-1f08-42b8-ac54-4d1aac4f81e8\") " pod="openshift-monitoring/node-exporter-6qp5m" Oct 31 00:56:27.689509 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.689398 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ccxz\" (UniqueName: \"kubernetes.io/projected/dd841d54-1f08-42b8-ac54-4d1aac4f81e8-kube-api-access-9ccxz\") pod \"node-exporter-6qp5m\" (UID: \"dd841d54-1f08-42b8-ac54-4d1aac4f81e8\") " pod="openshift-monitoring/node-exporter-6qp5m" Oct 31 00:56:27.689509 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.689435 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dd841d54-1f08-42b8-ac54-4d1aac4f81e8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6qp5m\" (UID: \"dd841d54-1f08-42b8-ac54-4d1aac4f81e8\") " pod="openshift-monitoring/node-exporter-6qp5m" Oct 31 00:56:27.689509 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.689473 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dd841d54-1f08-42b8-ac54-4d1aac4f81e8-node-exporter-tls\") pod \"node-exporter-6qp5m\" (UID: \"dd841d54-1f08-42b8-ac54-4d1aac4f81e8\") " pod="openshift-monitoring/node-exporter-6qp5m" Oct 31 00:56:27.689509 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.689487 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dd841d54-1f08-42b8-ac54-4d1aac4f81e8-root\") pod \"node-exporter-6qp5m\" (UID: \"dd841d54-1f08-42b8-ac54-4d1aac4f81e8\") " pod="openshift-monitoring/node-exporter-6qp5m" Oct 31 00:56:27.749678 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.749636 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:27.790357 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.790330 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dd841d54-1f08-42b8-ac54-4d1aac4f81e8-node-exporter-tls\") pod \"node-exporter-6qp5m\" (UID: \"dd841d54-1f08-42b8-ac54-4d1aac4f81e8\") " pod="openshift-monitoring/node-exporter-6qp5m" Oct 31 00:56:27.790540 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.790365 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dd841d54-1f08-42b8-ac54-4d1aac4f81e8-root\") pod \"node-exporter-6qp5m\" (UID: \"dd841d54-1f08-42b8-ac54-4d1aac4f81e8\") " pod="openshift-monitoring/node-exporter-6qp5m" Oct 31 00:56:27.790540 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.790397 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dd841d54-1f08-42b8-ac54-4d1aac4f81e8-node-exporter-textfile\") pod \"node-exporter-6qp5m\" (UID: \"dd841d54-1f08-42b8-ac54-4d1aac4f81e8\") " pod="openshift-monitoring/node-exporter-6qp5m" Oct 31 00:56:27.790540 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.790451 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dd841d54-1f08-42b8-ac54-4d1aac4f81e8-node-exporter-wtmp\") pod \"node-exporter-6qp5m\" (UID: \"dd841d54-1f08-42b8-ac54-4d1aac4f81e8\") " pod="openshift-monitoring/node-exporter-6qp5m" Oct 31 00:56:27.790540 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.790475 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dd841d54-1f08-42b8-ac54-4d1aac4f81e8-root\") pod \"node-exporter-6qp5m\" (UID: \"dd841d54-1f08-42b8-ac54-4d1aac4f81e8\") " pod="openshift-monitoring/node-exporter-6qp5m" Oct 31 00:56:27.790540 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.790479 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd841d54-1f08-42b8-ac54-4d1aac4f81e8-sys\") pod \"node-exporter-6qp5m\" (UID: \"dd841d54-1f08-42b8-ac54-4d1aac4f81e8\") " pod="openshift-monitoring/node-exporter-6qp5m" Oct 31 00:56:27.790540 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.790514 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd841d54-1f08-42b8-ac54-4d1aac4f81e8-sys\") pod \"node-exporter-6qp5m\" (UID: \"dd841d54-1f08-42b8-ac54-4d1aac4f81e8\") " pod="openshift-monitoring/node-exporter-6qp5m" Oct 31 00:56:27.790540 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.790524 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dd841d54-1f08-42b8-ac54-4d1aac4f81e8-node-exporter-accelerators-collector-config\") pod \"node-exporter-6qp5m\" (UID: \"dd841d54-1f08-42b8-ac54-4d1aac4f81e8\") " pod="openshift-monitoring/node-exporter-6qp5m" Oct 31 00:56:27.790857 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.790642 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dd841d54-1f08-42b8-ac54-4d1aac4f81e8-node-exporter-wtmp\") pod \"node-exporter-6qp5m\" (UID: \"dd841d54-1f08-42b8-ac54-4d1aac4f81e8\") " pod="openshift-monitoring/node-exporter-6qp5m" Oct 31 00:56:27.790857 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.790681 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd841d54-1f08-42b8-ac54-4d1aac4f81e8-metrics-client-ca\") pod \"node-exporter-6qp5m\" (UID: \"dd841d54-1f08-42b8-ac54-4d1aac4f81e8\") " pod="openshift-monitoring/node-exporter-6qp5m" Oct 31 00:56:27.790857 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.790706 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ccxz\" (UniqueName: \"kubernetes.io/projected/dd841d54-1f08-42b8-ac54-4d1aac4f81e8-kube-api-access-9ccxz\") pod \"node-exporter-6qp5m\" (UID: \"dd841d54-1f08-42b8-ac54-4d1aac4f81e8\") " pod="openshift-monitoring/node-exporter-6qp5m" Oct 31 00:56:27.790857 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.790741 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dd841d54-1f08-42b8-ac54-4d1aac4f81e8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6qp5m\" (UID: \"dd841d54-1f08-42b8-ac54-4d1aac4f81e8\") " pod="openshift-monitoring/node-exporter-6qp5m" Oct 31 00:56:27.791057 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.790855 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dd841d54-1f08-42b8-ac54-4d1aac4f81e8-node-exporter-textfile\") pod \"node-exporter-6qp5m\" (UID: \"dd841d54-1f08-42b8-ac54-4d1aac4f81e8\") " pod="openshift-monitoring/node-exporter-6qp5m" Oct 31 00:56:27.791124 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.791054 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dd841d54-1f08-42b8-ac54-4d1aac4f81e8-node-exporter-accelerators-collector-config\") pod \"node-exporter-6qp5m\" (UID: \"dd841d54-1f08-42b8-ac54-4d1aac4f81e8\") " pod="openshift-monitoring/node-exporter-6qp5m" Oct 31 00:56:27.791312 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.791269 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd841d54-1f08-42b8-ac54-4d1aac4f81e8-metrics-client-ca\") pod \"node-exporter-6qp5m\" (UID: \"dd841d54-1f08-42b8-ac54-4d1aac4f81e8\") " pod="openshift-monitoring/node-exporter-6qp5m" Oct 31 00:56:27.793360 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.793302 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dd841d54-1f08-42b8-ac54-4d1aac4f81e8-node-exporter-tls\") pod \"node-exporter-6qp5m\" (UID: \"dd841d54-1f08-42b8-ac54-4d1aac4f81e8\") " pod="openshift-monitoring/node-exporter-6qp5m" Oct 31 00:56:27.793476 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.793382 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dd841d54-1f08-42b8-ac54-4d1aac4f81e8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6qp5m\" (UID: \"dd841d54-1f08-42b8-ac54-4d1aac4f81e8\") " pod="openshift-monitoring/node-exporter-6qp5m" Oct 31 00:56:27.800636 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.800604 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ccxz\" (UniqueName: \"kubernetes.io/projected/dd841d54-1f08-42b8-ac54-4d1aac4f81e8-kube-api-access-9ccxz\") pod \"node-exporter-6qp5m\" (UID: \"dd841d54-1f08-42b8-ac54-4d1aac4f81e8\") " pod="openshift-monitoring/node-exporter-6qp5m" Oct 31 00:56:27.856978 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.856940 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:27.891762 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.891735 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6qp5m" Oct 31 00:56:27.922788 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:56:27.922760 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd841d54_1f08_42b8_ac54_4d1aac4f81e8.slice/crio-7e168232da0e5d7e8bad7a6aeb61584dcb4715570bc651f226b086ee06decb11 WatchSource:0}: Error finding container 7e168232da0e5d7e8bad7a6aeb61584dcb4715570bc651f226b086ee06decb11: Status 404 returned error can't find the container with id 7e168232da0e5d7e8bad7a6aeb61584dcb4715570bc651f226b086ee06decb11 Oct 31 00:56:27.974256 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:27.974224 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:28.147810 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.147730 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:28.604904 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.604806 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:28.674142 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.674108 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:56:28.696889 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.696864 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:56:28.697039 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.697028 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:28.702688 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.702664 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 00:56:28.756993 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.756965 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 00:56:28.756993 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.756965 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 00:56:28.757202 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.756998 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 00:56:28.757202 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.757015 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 00:56:28.757202 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.757036 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-l8qqk\"" Oct 31 00:56:28.757202 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.757044 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 00:56:28.757202 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.757157 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 00:56:28.757202 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.757190 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 00:56:28.757562 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.757247 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 00:56:28.798438 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.798395 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:28.798597 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.798444 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxjnf\" (UniqueName: \"kubernetes.io/projected/61debc59-fba7-4053-b2e5-e70feb446400-kube-api-access-dxjnf\") pod \"alertmanager-main-0\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:28.798597 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.798542 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61debc59-fba7-4053-b2e5-e70feb446400-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:28.798597 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.798577 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-config-volume\") pod \"alertmanager-main-0\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:28.798758 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.798620 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:28.798758 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.798661 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61debc59-fba7-4053-b2e5-e70feb446400-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:28.798758 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.798696 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/61debc59-fba7-4053-b2e5-e70feb446400-tls-assets\") pod \"alertmanager-main-0\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:28.798758 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.798719 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/61debc59-fba7-4053-b2e5-e70feb446400-config-out\") pod \"alertmanager-main-0\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:28.798758 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.798745 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:28.798998 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.798774 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:28.798998 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.798818 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:28.798998 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.798841 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-web-config\") pod \"alertmanager-main-0\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:28.798998 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.798874 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/61debc59-fba7-4053-b2e5-e70feb446400-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:28.841414 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.841384 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:28.900102 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.900035 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:28.900102 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.900073 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-web-config\") pod \"alertmanager-main-0\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:28.900102 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.900101 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/61debc59-fba7-4053-b2e5-e70feb446400-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:28.900387 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.900139 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:28.900387 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.900159 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxjnf\" (UniqueName: \"kubernetes.io/projected/61debc59-fba7-4053-b2e5-e70feb446400-kube-api-access-dxjnf\") pod \"alertmanager-main-0\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:28.900387 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.900187 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61debc59-fba7-4053-b2e5-e70feb446400-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:28.900387 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.900209 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-config-volume\") pod \"alertmanager-main-0\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:28.900387 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.900238 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:28.900387 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.900269 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61debc59-fba7-4053-b2e5-e70feb446400-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:28.900387 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.900299 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/61debc59-fba7-4053-b2e5-e70feb446400-tls-assets\") pod \"alertmanager-main-0\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:28.900387 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.900313 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/61debc59-fba7-4053-b2e5-e70feb446400-config-out\") pod \"alertmanager-main-0\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:28.900387 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.900340 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:28.900387 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.900370 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:28.903224 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.903189 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61debc59-fba7-4053-b2e5-e70feb446400-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:28.903543 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.903512 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/61debc59-fba7-4053-b2e5-e70feb446400-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:28.904128 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.904107 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:28.904268 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.904249 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61debc59-fba7-4053-b2e5-e70feb446400-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:28.904848 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.904805 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:28.905017 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.904957 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:28.905017 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.905007 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-config-volume\") pod \"alertmanager-main-0\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:28.907946 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.907914 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/61debc59-fba7-4053-b2e5-e70feb446400-config-out\") pod \"alertmanager-main-0\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:28.917810 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.917792 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-web-config\") pod \"alertmanager-main-0\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:28.918144 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.918122 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:28.918832 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.918746 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/61debc59-fba7-4053-b2e5-e70feb446400-tls-assets\") pod \"alertmanager-main-0\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:28.918832 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.918809 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:28.921738 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.921714 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6qp5m" event={"ID":"dd841d54-1f08-42b8-ac54-4d1aac4f81e8","Type":"ContainerStarted","Data":"7e168232da0e5d7e8bad7a6aeb61584dcb4715570bc651f226b086ee06decb11"} Oct 31 00:56:28.923305 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:28.923284 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxjnf\" (UniqueName: \"kubernetes.io/projected/61debc59-fba7-4053-b2e5-e70feb446400-kube-api-access-dxjnf\") pod \"alertmanager-main-0\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:29.104380 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:29.104344 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:29.470985 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:29.470908 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:56:29.490982 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:56:29.490949 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61debc59_fba7_4053_b2e5_e70feb446400.slice/crio-084bb1980578b9463f2e91e69b5efbad2bbea31e9932c5a13f27ef1a6e97f9d9 WatchSource:0}: Error finding container 084bb1980578b9463f2e91e69b5efbad2bbea31e9932c5a13f27ef1a6e97f9d9: Status 404 returned error can't find the container with id 084bb1980578b9463f2e91e69b5efbad2bbea31e9932c5a13f27ef1a6e97f9d9 Oct 31 00:56:29.925789 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:29.925753 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:29.926256 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:29.926190 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"61debc59-fba7-4053-b2e5-e70feb446400","Type":"ContainerStarted","Data":"084bb1980578b9463f2e91e69b5efbad2bbea31e9932c5a13f27ef1a6e97f9d9"} Oct 31 00:56:29.928353 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:29.928062 4440 generic.go:356] "Generic (PLEG): container finished" podID="dd841d54-1f08-42b8-ac54-4d1aac4f81e8" containerID="db3bd6944ad30ecc971e6377f70b86fbcd84840f13ed3beb02bd70ea9cc1f3b7" exitCode=0 Oct 31 00:56:29.928353 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:29.928155 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6qp5m" event={"ID":"dd841d54-1f08-42b8-ac54-4d1aac4f81e8","Type":"ContainerDied","Data":"db3bd6944ad30ecc971e6377f70b86fbcd84840f13ed3beb02bd70ea9cc1f3b7"} Oct 31 00:56:30.071683 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:30.071614 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:30.569969 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:30.569889 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:30.784952 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:30.784914 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:30.934703 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:30.934667 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6qp5m" event={"ID":"dd841d54-1f08-42b8-ac54-4d1aac4f81e8","Type":"ContainerStarted","Data":"069cd41b1f6b7836e131129a944d8dfd8c3447b6480e9343af9a1cf8c26a0727"} Oct 31 00:56:30.935157 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:30.934733 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6qp5m" event={"ID":"dd841d54-1f08-42b8-ac54-4d1aac4f81e8","Type":"ContainerStarted","Data":"71546ac95641e004acf74208a4f20dae74b5ca20b659bb9d81e5350f1f10af48"} Oct 31 00:56:30.957159 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:30.957096 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6qp5m" podStartSLOduration=2.262168804 podStartE2EDuration="3.957079131s" totalImagesPullingTime="1.595426765s" totalInitContainerRuntime="99.483562ms" isStatefulPod=true podCreationTimestamp="2026-10-31 00:56:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:30.956065299 +0000 UTC m=+35.314787629" watchObservedRunningTime="2026-10-31 00:56:30.957079131 +0000 UTC m=+35.315801448" Oct 31 00:56:31.004053 ip-10-0-130-39 kubenswrapper[4440]: E1031 00:56:31.004015 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 00:56:31.643612 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:31.643531 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:31.831489 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:31.831454 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:31.940298 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:31.940261 4440 generic.go:356] "Generic (PLEG): container finished" podID="61debc59-fba7-4053-b2e5-e70feb446400" containerID="a0d2691c59016a972c4acac6ab9f35e8abde11a60112c8900192ccff5710ef21" exitCode=0 Oct 31 00:56:31.940712 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:31.940346 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"61debc59-fba7-4053-b2e5-e70feb446400","Type":"ContainerDied","Data":"a0d2691c59016a972c4acac6ab9f35e8abde11a60112c8900192ccff5710ef21"} Oct 31 00:56:32.324475 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:32.324374 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5xvng" Oct 31 00:56:32.368086 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:32.367984 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:32.533908 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:32.533866 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:33.892050 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:33.891988 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:33.951361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:33.951326 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-gjjhv" event={"ID":"6ed6c333-706a-44dc-aae6-54be82855800","Type":"ContainerStarted","Data":"3cefbb9576bacd29042303b8126ebd202dbc7d67a50071d4e576f2119ee30cae"} Oct 31 00:56:33.953747 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:33.953727 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-gjjhv" Oct 31 00:56:33.959443 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:33.959036 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"61debc59-fba7-4053-b2e5-e70feb446400","Type":"ContainerStarted","Data":"618ffb8a6bb1004ad87b17d44abd3fd5df72a45ae7d0741ce1f0a9b6fb531bbb"} Oct 31 00:56:33.959443 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:33.959091 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"61debc59-fba7-4053-b2e5-e70feb446400","Type":"ContainerStarted","Data":"1a729db225f690dd4323f6e84ce9a61f262c9e1c819b407ea10288ad2240600d"} Oct 31 00:56:33.959443 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:33.959140 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"61debc59-fba7-4053-b2e5-e70feb446400","Type":"ContainerStarted","Data":"5b82a2b49bfe6f24f5cc7a611e5b5ac3f7b469d3beaa3c37832ddea136eee130"} Oct 31 00:56:33.959443 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:33.959175 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"61debc59-fba7-4053-b2e5-e70feb446400","Type":"ContainerStarted","Data":"7e09a500742a4c1af5261cd5276860040174d50f9f990435df890960773176aa"} Oct 31 00:56:33.961637 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:33.961615 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-gjjhv" Oct 31 00:56:33.974980 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:33.974914 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-gjjhv" podStartSLOduration=1.763976749 podStartE2EDuration="18.974904949s" totalImagesPullingTime="17.2109282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:33.973367147 +0000 UTC m=+38.332089462" watchObservedRunningTime="2026-10-31 00:56:33.974904949 +0000 UTC m=+38.333627264" Oct 31 00:56:34.052616 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:34.052581 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:34.529928 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:34.529888 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:34.657565 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:34.657522 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:34.966490 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:34.966454 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"61debc59-fba7-4053-b2e5-e70feb446400","Type":"ContainerStarted","Data":"f948c1ec087dd12d18a914f0f188f509bcf4428fca8d9c640253f2e93bd622f0"} Oct 31 00:56:35.286456 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:35.286353 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5999b68fb8-9hrhl"] Oct 31 00:56:35.302656 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:35.302621 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5999b68fb8-9hrhl"] Oct 31 00:56:35.302822 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:35.302744 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5999b68fb8-9hrhl" Oct 31 00:56:35.307616 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:35.307588 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 31 00:56:35.307750 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:35.307669 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 31 00:56:35.307750 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:35.307680 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 31 00:56:35.311436 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:35.311392 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 31 00:56:35.360327 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:35.360295 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 31 00:56:35.360560 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:35.360399 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-55222\"" Oct 31 00:56:35.360560 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:35.360525 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 31 00:56:35.461491 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:35.461463 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d8e4bad-7966-4577-b142-c3595de3500a-oauth-serving-cert\") pod \"console-5999b68fb8-9hrhl\" (UID: \"9d8e4bad-7966-4577-b142-c3595de3500a\") " pod="openshift-console/console-5999b68fb8-9hrhl" Oct 31 00:56:35.461652 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:35.461502 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8e4bad-7966-4577-b142-c3595de3500a-service-ca\") pod \"console-5999b68fb8-9hrhl\" (UID: \"9d8e4bad-7966-4577-b142-c3595de3500a\") " pod="openshift-console/console-5999b68fb8-9hrhl" Oct 31 00:56:35.461652 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:35.461521 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbcn9\" (UniqueName: \"kubernetes.io/projected/9d8e4bad-7966-4577-b142-c3595de3500a-kube-api-access-gbcn9\") pod \"console-5999b68fb8-9hrhl\" (UID: \"9d8e4bad-7966-4577-b142-c3595de3500a\") " pod="openshift-console/console-5999b68fb8-9hrhl" Oct 31 00:56:35.461652 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:35.461604 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d8e4bad-7966-4577-b142-c3595de3500a-tmp\") pod \"console-5999b68fb8-9hrhl\" (UID: \"9d8e4bad-7966-4577-b142-c3595de3500a\") " pod="openshift-console/console-5999b68fb8-9hrhl" Oct 31 00:56:35.461652 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:35.461646 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d8e4bad-7966-4577-b142-c3595de3500a-console-config\") pod \"console-5999b68fb8-9hrhl\" (UID: \"9d8e4bad-7966-4577-b142-c3595de3500a\") " pod="openshift-console/console-5999b68fb8-9hrhl" Oct 31 00:56:35.461842 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:35.461718 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d8e4bad-7966-4577-b142-c3595de3500a-console-oauth-config\") pod \"console-5999b68fb8-9hrhl\" (UID: \"9d8e4bad-7966-4577-b142-c3595de3500a\") " pod="openshift-console/console-5999b68fb8-9hrhl" Oct 31 00:56:35.461842 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:35.461799 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d8e4bad-7966-4577-b142-c3595de3500a-trusted-ca-bundle\") pod \"console-5999b68fb8-9hrhl\" (UID: \"9d8e4bad-7966-4577-b142-c3595de3500a\") " pod="openshift-console/console-5999b68fb8-9hrhl" Oct 31 00:56:35.461938 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:35.461859 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d8e4bad-7966-4577-b142-c3595de3500a-console-serving-cert\") pod \"console-5999b68fb8-9hrhl\" (UID: \"9d8e4bad-7966-4577-b142-c3595de3500a\") " pod="openshift-console/console-5999b68fb8-9hrhl" Oct 31 00:56:35.568885 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:35.568796 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d8e4bad-7966-4577-b142-c3595de3500a-oauth-serving-cert\") pod \"console-5999b68fb8-9hrhl\" (UID: \"9d8e4bad-7966-4577-b142-c3595de3500a\") " pod="openshift-console/console-5999b68fb8-9hrhl" Oct 31 00:56:35.568885 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:35.568854 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8e4bad-7966-4577-b142-c3595de3500a-service-ca\") pod \"console-5999b68fb8-9hrhl\" (UID: \"9d8e4bad-7966-4577-b142-c3595de3500a\") " pod="openshift-console/console-5999b68fb8-9hrhl" Oct 31 00:56:35.568885 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:35.568880 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbcn9\" (UniqueName: \"kubernetes.io/projected/9d8e4bad-7966-4577-b142-c3595de3500a-kube-api-access-gbcn9\") pod \"console-5999b68fb8-9hrhl\" (UID: \"9d8e4bad-7966-4577-b142-c3595de3500a\") " pod="openshift-console/console-5999b68fb8-9hrhl" Oct 31 00:56:35.569174 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:35.568905 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d8e4bad-7966-4577-b142-c3595de3500a-tmp\") pod \"console-5999b68fb8-9hrhl\" (UID: \"9d8e4bad-7966-4577-b142-c3595de3500a\") " pod="openshift-console/console-5999b68fb8-9hrhl" Oct 31 00:56:35.569174 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:35.568929 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d8e4bad-7966-4577-b142-c3595de3500a-console-config\") pod \"console-5999b68fb8-9hrhl\" (UID: \"9d8e4bad-7966-4577-b142-c3595de3500a\") " pod="openshift-console/console-5999b68fb8-9hrhl" Oct 31 00:56:35.569174 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:35.568957 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d8e4bad-7966-4577-b142-c3595de3500a-console-oauth-config\") pod \"console-5999b68fb8-9hrhl\" (UID: \"9d8e4bad-7966-4577-b142-c3595de3500a\") " pod="openshift-console/console-5999b68fb8-9hrhl" Oct 31 00:56:35.569174 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:35.569000 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d8e4bad-7966-4577-b142-c3595de3500a-trusted-ca-bundle\") pod \"console-5999b68fb8-9hrhl\" (UID: \"9d8e4bad-7966-4577-b142-c3595de3500a\") " pod="openshift-console/console-5999b68fb8-9hrhl" Oct 31 00:56:35.569174 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:35.569044 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d8e4bad-7966-4577-b142-c3595de3500a-console-serving-cert\") pod \"console-5999b68fb8-9hrhl\" (UID: \"9d8e4bad-7966-4577-b142-c3595de3500a\") " pod="openshift-console/console-5999b68fb8-9hrhl" Oct 31 00:56:35.569891 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:35.569759 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8e4bad-7966-4577-b142-c3595de3500a-service-ca\") pod \"console-5999b68fb8-9hrhl\" (UID: \"9d8e4bad-7966-4577-b142-c3595de3500a\") " pod="openshift-console/console-5999b68fb8-9hrhl" Oct 31 00:56:35.569891 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:35.569838 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d8e4bad-7966-4577-b142-c3595de3500a-tmp\") pod \"console-5999b68fb8-9hrhl\" (UID: \"9d8e4bad-7966-4577-b142-c3595de3500a\") " pod="openshift-console/console-5999b68fb8-9hrhl" Oct 31 00:56:35.570102 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:35.570076 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d8e4bad-7966-4577-b142-c3595de3500a-console-config\") pod \"console-5999b68fb8-9hrhl\" (UID: \"9d8e4bad-7966-4577-b142-c3595de3500a\") " pod="openshift-console/console-5999b68fb8-9hrhl" Oct 31 00:56:35.570336 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:35.570279 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d8e4bad-7966-4577-b142-c3595de3500a-trusted-ca-bundle\") pod \"console-5999b68fb8-9hrhl\" (UID: \"9d8e4bad-7966-4577-b142-c3595de3500a\") " pod="openshift-console/console-5999b68fb8-9hrhl" Oct 31 00:56:35.570581 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:35.570560 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d8e4bad-7966-4577-b142-c3595de3500a-oauth-serving-cert\") pod \"console-5999b68fb8-9hrhl\" (UID: \"9d8e4bad-7966-4577-b142-c3595de3500a\") " pod="openshift-console/console-5999b68fb8-9hrhl" Oct 31 00:56:35.571941 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:35.571916 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d8e4bad-7966-4577-b142-c3595de3500a-console-serving-cert\") pod \"console-5999b68fb8-9hrhl\" (UID: \"9d8e4bad-7966-4577-b142-c3595de3500a\") " pod="openshift-console/console-5999b68fb8-9hrhl" Oct 31 00:56:35.572091 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:35.571974 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d8e4bad-7966-4577-b142-c3595de3500a-console-oauth-config\") pod \"console-5999b68fb8-9hrhl\" (UID: \"9d8e4bad-7966-4577-b142-c3595de3500a\") " pod="openshift-console/console-5999b68fb8-9hrhl" Oct 31 00:56:35.577392 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:35.577369 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbcn9\" (UniqueName: \"kubernetes.io/projected/9d8e4bad-7966-4577-b142-c3595de3500a-kube-api-access-gbcn9\") pod \"console-5999b68fb8-9hrhl\" (UID: \"9d8e4bad-7966-4577-b142-c3595de3500a\") " pod="openshift-console/console-5999b68fb8-9hrhl" Oct 31 00:56:35.591467 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:35.591437 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:35.624444 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:35.624391 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5999b68fb8-9hrhl" Oct 31 00:56:35.728714 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:35.724471 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:35.953239 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:35.953213 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5999b68fb8-9hrhl"] Oct 31 00:56:35.955406 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:56:35.955379 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d8e4bad_7966_4577_b142_c3595de3500a.slice/crio-e750f64baf575b1eea72cb43889b60b59961a990dcd8cbdcb7643d27b561554d WatchSource:0}: Error finding container e750f64baf575b1eea72cb43889b60b59961a990dcd8cbdcb7643d27b561554d: Status 404 returned error can't find the container with id e750f64baf575b1eea72cb43889b60b59961a990dcd8cbdcb7643d27b561554d Oct 31 00:56:35.970969 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:35.970937 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5999b68fb8-9hrhl" event={"ID":"9d8e4bad-7966-4577-b142-c3595de3500a","Type":"ContainerStarted","Data":"e750f64baf575b1eea72cb43889b60b59961a990dcd8cbdcb7643d27b561554d"} Oct 31 00:56:35.976146 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:35.976120 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"61debc59-fba7-4053-b2e5-e70feb446400","Type":"ContainerStarted","Data":"83f462d6cfd5698fe2cb06710d1949b684bcef5a765923d5f1a970c5e7444fbe"} Oct 31 00:56:36.007862 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:36.007303 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.246340913 podStartE2EDuration="8.007286168s" totalImagesPullingTime="4.738956998s" totalInitContainerRuntime="21.988257ms" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:36.005389465 +0000 UTC m=+40.364111794" watchObservedRunningTime="2026-10-31 00:56:36.007286168 +0000 UTC m=+40.366008484" Oct 31 00:56:36.639448 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:36.638957 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:36.763440 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:36.759708 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:37.405388 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:37.405304 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:37.537449 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:37.537378 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:37.850781 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:37.850746 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:37.884355 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:37.884322 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-85c74bc6cd-ggkcd" Oct 31 00:56:38.012022 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:38.011987 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:38.159959 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:38.159852 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:38.311252 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:38.311215 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:41.001241 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:41.001203 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5999b68fb8-9hrhl" event={"ID":"9d8e4bad-7966-4577-b142-c3595de3500a","Type":"ContainerStarted","Data":"9b66b09004f4a0f777c05290c9cb38409b91600ecdc32bb6fa39476e107aeee8"} Oct 31 00:56:41.028659 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:41.028599 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5999b68fb8-9hrhl" podStartSLOduration=2.06503067 podStartE2EDuration="6.028582637s" totalImagesPullingTime="3.963551967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:41.026840313 +0000 UTC m=+45.385562649" watchObservedRunningTime="2026-10-31 00:56:41.028582637 +0000 UTC m=+45.387304953" Oct 31 00:56:41.637049 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:41.637010 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5999b68fb8-9hrhl"] Oct 31 00:56:41.682651 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:41.682614 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-57d44c567c-fs4wp"] Oct 31 00:56:41.717970 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:41.717919 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57d44c567c-fs4wp"] Oct 31 00:56:41.717970 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:41.717958 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57d44c567c-fs4wp" Oct 31 00:56:41.794112 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:41.794079 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:41.834359 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:41.834329 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bc93122-80c6-435a-9b02-c32d2da99d5d-tmp\") pod \"console-57d44c567c-fs4wp\" (UID: \"4bc93122-80c6-435a-9b02-c32d2da99d5d\") " pod="openshift-console/console-57d44c567c-fs4wp" Oct 31 00:56:41.834359 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:41.834364 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4bc93122-80c6-435a-9b02-c32d2da99d5d-oauth-serving-cert\") pod \"console-57d44c567c-fs4wp\" (UID: \"4bc93122-80c6-435a-9b02-c32d2da99d5d\") " pod="openshift-console/console-57d44c567c-fs4wp" Oct 31 00:56:41.834567 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:41.834384 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bc93122-80c6-435a-9b02-c32d2da99d5d-trusted-ca-bundle\") pod \"console-57d44c567c-fs4wp\" (UID: \"4bc93122-80c6-435a-9b02-c32d2da99d5d\") " pod="openshift-console/console-57d44c567c-fs4wp" Oct 31 00:56:41.834567 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:41.834404 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4bc93122-80c6-435a-9b02-c32d2da99d5d-console-oauth-config\") pod \"console-57d44c567c-fs4wp\" (UID: \"4bc93122-80c6-435a-9b02-c32d2da99d5d\") " pod="openshift-console/console-57d44c567c-fs4wp" Oct 31 00:56:41.834567 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:41.834480 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4bc93122-80c6-435a-9b02-c32d2da99d5d-console-config\") pod \"console-57d44c567c-fs4wp\" (UID: \"4bc93122-80c6-435a-9b02-c32d2da99d5d\") " pod="openshift-console/console-57d44c567c-fs4wp" Oct 31 00:56:41.834567 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:41.834546 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnrd7\" (UniqueName: \"kubernetes.io/projected/4bc93122-80c6-435a-9b02-c32d2da99d5d-kube-api-access-lnrd7\") pod \"console-57d44c567c-fs4wp\" (UID: \"4bc93122-80c6-435a-9b02-c32d2da99d5d\") " pod="openshift-console/console-57d44c567c-fs4wp" Oct 31 00:56:41.834694 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:41.834591 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4bc93122-80c6-435a-9b02-c32d2da99d5d-console-serving-cert\") pod \"console-57d44c567c-fs4wp\" (UID: \"4bc93122-80c6-435a-9b02-c32d2da99d5d\") " pod="openshift-console/console-57d44c567c-fs4wp" Oct 31 00:56:41.834694 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:41.834609 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4bc93122-80c6-435a-9b02-c32d2da99d5d-service-ca\") pod \"console-57d44c567c-fs4wp\" (UID: \"4bc93122-80c6-435a-9b02-c32d2da99d5d\") " pod="openshift-console/console-57d44c567c-fs4wp" Oct 31 00:56:41.919440 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:41.919389 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:41.935503 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:41.935474 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4bc93122-80c6-435a-9b02-c32d2da99d5d-console-serving-cert\") pod \"console-57d44c567c-fs4wp\" (UID: \"4bc93122-80c6-435a-9b02-c32d2da99d5d\") " pod="openshift-console/console-57d44c567c-fs4wp" Oct 31 00:56:41.935597 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:41.935508 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4bc93122-80c6-435a-9b02-c32d2da99d5d-service-ca\") pod \"console-57d44c567c-fs4wp\" (UID: \"4bc93122-80c6-435a-9b02-c32d2da99d5d\") " pod="openshift-console/console-57d44c567c-fs4wp" Oct 31 00:56:41.935597 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:41.935542 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bc93122-80c6-435a-9b02-c32d2da99d5d-tmp\") pod \"console-57d44c567c-fs4wp\" (UID: \"4bc93122-80c6-435a-9b02-c32d2da99d5d\") " pod="openshift-console/console-57d44c567c-fs4wp" Oct 31 00:56:41.935597 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:41.935571 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4bc93122-80c6-435a-9b02-c32d2da99d5d-oauth-serving-cert\") pod \"console-57d44c567c-fs4wp\" (UID: \"4bc93122-80c6-435a-9b02-c32d2da99d5d\") " pod="openshift-console/console-57d44c567c-fs4wp" Oct 31 00:56:41.935733 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:41.935596 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bc93122-80c6-435a-9b02-c32d2da99d5d-trusted-ca-bundle\") pod \"console-57d44c567c-fs4wp\" (UID: \"4bc93122-80c6-435a-9b02-c32d2da99d5d\") " pod="openshift-console/console-57d44c567c-fs4wp" Oct 31 00:56:41.935733 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:41.935622 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4bc93122-80c6-435a-9b02-c32d2da99d5d-console-oauth-config\") pod \"console-57d44c567c-fs4wp\" (UID: \"4bc93122-80c6-435a-9b02-c32d2da99d5d\") " pod="openshift-console/console-57d44c567c-fs4wp" Oct 31 00:56:41.935733 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:41.935643 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4bc93122-80c6-435a-9b02-c32d2da99d5d-console-config\") pod \"console-57d44c567c-fs4wp\" (UID: \"4bc93122-80c6-435a-9b02-c32d2da99d5d\") " pod="openshift-console/console-57d44c567c-fs4wp" Oct 31 00:56:41.935733 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:41.935680 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnrd7\" (UniqueName: \"kubernetes.io/projected/4bc93122-80c6-435a-9b02-c32d2da99d5d-kube-api-access-lnrd7\") pod \"console-57d44c567c-fs4wp\" (UID: \"4bc93122-80c6-435a-9b02-c32d2da99d5d\") " pod="openshift-console/console-57d44c567c-fs4wp" Oct 31 00:56:41.936153 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:41.936132 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bc93122-80c6-435a-9b02-c32d2da99d5d-tmp\") pod \"console-57d44c567c-fs4wp\" (UID: \"4bc93122-80c6-435a-9b02-c32d2da99d5d\") " pod="openshift-console/console-57d44c567c-fs4wp" Oct 31 00:56:41.936372 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:41.936351 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4bc93122-80c6-435a-9b02-c32d2da99d5d-service-ca\") pod \"console-57d44c567c-fs4wp\" (UID: \"4bc93122-80c6-435a-9b02-c32d2da99d5d\") " pod="openshift-console/console-57d44c567c-fs4wp" Oct 31 00:56:41.936550 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:41.936533 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4bc93122-80c6-435a-9b02-c32d2da99d5d-console-config\") pod \"console-57d44c567c-fs4wp\" (UID: \"4bc93122-80c6-435a-9b02-c32d2da99d5d\") " pod="openshift-console/console-57d44c567c-fs4wp" Oct 31 00:56:41.936647 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:41.936625 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4bc93122-80c6-435a-9b02-c32d2da99d5d-oauth-serving-cert\") pod \"console-57d44c567c-fs4wp\" (UID: \"4bc93122-80c6-435a-9b02-c32d2da99d5d\") " pod="openshift-console/console-57d44c567c-fs4wp" Oct 31 00:56:41.936690 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:41.936641 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bc93122-80c6-435a-9b02-c32d2da99d5d-trusted-ca-bundle\") pod \"console-57d44c567c-fs4wp\" (UID: \"4bc93122-80c6-435a-9b02-c32d2da99d5d\") " pod="openshift-console/console-57d44c567c-fs4wp" Oct 31 00:56:41.938133 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:41.938111 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4bc93122-80c6-435a-9b02-c32d2da99d5d-console-serving-cert\") pod \"console-57d44c567c-fs4wp\" (UID: \"4bc93122-80c6-435a-9b02-c32d2da99d5d\") " pod="openshift-console/console-57d44c567c-fs4wp" Oct 31 00:56:41.938133 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:41.938120 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4bc93122-80c6-435a-9b02-c32d2da99d5d-console-oauth-config\") pod \"console-57d44c567c-fs4wp\" (UID: \"4bc93122-80c6-435a-9b02-c32d2da99d5d\") " pod="openshift-console/console-57d44c567c-fs4wp" Oct 31 00:56:41.945482 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:41.945461 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnrd7\" (UniqueName: \"kubernetes.io/projected/4bc93122-80c6-435a-9b02-c32d2da99d5d-kube-api-access-lnrd7\") pod \"console-57d44c567c-fs4wp\" (UID: \"4bc93122-80c6-435a-9b02-c32d2da99d5d\") " pod="openshift-console/console-57d44c567c-fs4wp" Oct 31 00:56:42.041620 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:42.041583 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57d44c567c-fs4wp" Oct 31 00:56:42.342878 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:42.342804 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57d44c567c-fs4wp"] Oct 31 00:56:42.345840 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:56:42.345812 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bc93122_80c6_435a_9b02_c32d2da99d5d.slice/crio-94d2d9dbbb5a7e2d6abb30a4127cec23b258d1f9ffd28298d0154d278c6e57d0 WatchSource:0}: Error finding container 94d2d9dbbb5a7e2d6abb30a4127cec23b258d1f9ffd28298d0154d278c6e57d0: Status 404 returned error can't find the container with id 94d2d9dbbb5a7e2d6abb30a4127cec23b258d1f9ffd28298d0154d278c6e57d0 Oct 31 00:56:43.009627 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:43.009581 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57d44c567c-fs4wp" event={"ID":"4bc93122-80c6-435a-9b02-c32d2da99d5d","Type":"ContainerStarted","Data":"ebfabe8808791fca1fa7ec1df9fa57e760beacf1801fec8cc44523e3046a93b2"} Oct 31 00:56:43.009893 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:43.009665 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57d44c567c-fs4wp" event={"ID":"4bc93122-80c6-435a-9b02-c32d2da99d5d","Type":"ContainerStarted","Data":"94d2d9dbbb5a7e2d6abb30a4127cec23b258d1f9ffd28298d0154d278c6e57d0"} Oct 31 00:56:43.032706 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:43.032650 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-57d44c567c-fs4wp" podStartSLOduration=2.032635712 podStartE2EDuration="2.032635712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:43.031945731 +0000 UTC m=+47.390668046" watchObservedRunningTime="2026-10-31 00:56:43.032635712 +0000 UTC m=+47.391358025" Oct 31 00:56:43.589212 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:43.589166 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:43.733557 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:43.733512 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:45.105063 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:45.105029 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:45.267678 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:45.267643 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:45.625466 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:45.625404 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5999b68fb8-9hrhl" Oct 31 00:56:48.861957 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:48.861915 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:48.973563 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:48.973534 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:52.042596 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:52.042567 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57d44c567c-fs4wp" Oct 31 00:56:52.043036 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:52.042621 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57d44c567c-fs4wp" Oct 31 00:56:52.047382 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:52.047359 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57d44c567c-fs4wp" Oct 31 00:56:53.047138 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:53.047101 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57d44c567c-fs4wp" Oct 31 00:56:53.796242 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:53.796207 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:55.220351 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:55.220314 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:55.373481 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:55.373446 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:55.860984 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:55.860951 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:55.983276 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:55.983246 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:57.651700 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:57.651649 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:57.771087 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:57.771053 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:58.039122 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.039090 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:56:58.039403 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.039380 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="61debc59-fba7-4053-b2e5-e70feb446400" containerName="alertmanager" containerID="cri-o://7e09a500742a4c1af5261cd5276860040174d50f9f990435df890960773176aa" gracePeriod=120 Oct 31 00:56:58.039499 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.039468 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="61debc59-fba7-4053-b2e5-e70feb446400" containerName="kube-rbac-proxy-metric" containerID="cri-o://f948c1ec087dd12d18a914f0f188f509bcf4428fca8d9c640253f2e93bd622f0" gracePeriod=120 Oct 31 00:56:58.039555 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.039479 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="61debc59-fba7-4053-b2e5-e70feb446400" containerName="kube-rbac-proxy-web" containerID="cri-o://1a729db225f690dd4323f6e84ce9a61f262c9e1c819b407ea10288ad2240600d" gracePeriod=120 Oct 31 00:56:58.039555 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.039497 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="61debc59-fba7-4053-b2e5-e70feb446400" containerName="kube-rbac-proxy" containerID="cri-o://618ffb8a6bb1004ad87b17d44abd3fd5df72a45ae7d0741ce1f0a9b6fb531bbb" gracePeriod=120 Oct 31 00:56:58.039641 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.039534 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="61debc59-fba7-4053-b2e5-e70feb446400" containerName="prom-label-proxy" containerID="cri-o://83f462d6cfd5698fe2cb06710d1949b684bcef5a765923d5f1a970c5e7444fbe" gracePeriod=120 Oct 31 00:56:58.039641 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.039480 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="61debc59-fba7-4053-b2e5-e70feb446400" containerName="config-reloader" containerID="cri-o://5b82a2b49bfe6f24f5cc7a611e5b5ac3f7b469d3beaa3c37832ddea136eee130" gracePeriod=120 Oct 31 00:56:58.053545 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.053524 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:58.171963 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.171921 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:58.321961 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.321903 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:58.351877 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.351843 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61debc59-fba7-4053-b2e5-e70feb446400-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61debc59-fba7-4053-b2e5-e70feb446400-metrics-client-ca\") pod \"61debc59-fba7-4053-b2e5-e70feb446400\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " Oct 31 00:56:58.352003 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.351896 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-secret-alertmanager-kube-rbac-proxy-metric\") pod \"61debc59-fba7-4053-b2e5-e70feb446400\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " Oct 31 00:56:58.352003 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.351933 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61debc59-fba7-4053-b2e5-e70feb446400-kube-api-access-dxjnf\" (UniqueName: \"kubernetes.io/projected/61debc59-fba7-4053-b2e5-e70feb446400-kube-api-access-dxjnf\") pod \"61debc59-fba7-4053-b2e5-e70feb446400\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " Oct 31 00:56:58.352003 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.351962 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-secret-alertmanager-kube-rbac-proxy-web\") pod \"61debc59-fba7-4053-b2e5-e70feb446400\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " Oct 31 00:56:58.352136 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.352005 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-secret-alertmanager-main-tls\") pod \"61debc59-fba7-4053-b2e5-e70feb446400\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " Oct 31 00:56:58.352136 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.352042 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-cluster-tls-config\") pod \"61debc59-fba7-4053-b2e5-e70feb446400\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " Oct 31 00:56:58.352136 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.352070 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-secret-alertmanager-kube-rbac-proxy\") pod \"61debc59-fba7-4053-b2e5-e70feb446400\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " Oct 31 00:56:58.352136 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.352117 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61debc59-fba7-4053-b2e5-e70feb446400-tls-assets\" (UniqueName: \"kubernetes.io/projected/61debc59-fba7-4053-b2e5-e70feb446400-tls-assets\") pod \"61debc59-fba7-4053-b2e5-e70feb446400\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " Oct 31 00:56:58.352322 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.352149 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61debc59-fba7-4053-b2e5-e70feb446400-config-out\" (UniqueName: \"kubernetes.io/empty-dir/61debc59-fba7-4053-b2e5-e70feb446400-config-out\") pod \"61debc59-fba7-4053-b2e5-e70feb446400\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " Oct 31 00:56:58.352322 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.352205 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61debc59-fba7-4053-b2e5-e70feb446400-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61debc59-fba7-4053-b2e5-e70feb446400-alertmanager-trusted-ca-bundle\") pod \"61debc59-fba7-4053-b2e5-e70feb446400\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " Oct 31 00:56:58.352322 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.352237 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-config-volume\" (UniqueName: \"kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-config-volume\") pod \"61debc59-fba7-4053-b2e5-e70feb446400\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " Oct 31 00:56:58.352322 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.352287 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61debc59-fba7-4053-b2e5-e70feb446400-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/61debc59-fba7-4053-b2e5-e70feb446400-alertmanager-main-db\") pod \"61debc59-fba7-4053-b2e5-e70feb446400\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " Oct 31 00:56:58.352322 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.352314 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-web-config\" (UniqueName: \"kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-web-config\") pod \"61debc59-fba7-4053-b2e5-e70feb446400\" (UID: \"61debc59-fba7-4053-b2e5-e70feb446400\") " Oct 31 00:56:58.352547 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.352339 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61debc59-fba7-4053-b2e5-e70feb446400-metrics-client-ca" pod "61debc59-fba7-4053-b2e5-e70feb446400" (UID: "61debc59-fba7-4053-b2e5-e70feb446400"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:56:58.352601 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.352558 4440 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61debc59-fba7-4053-b2e5-e70feb446400-metrics-client-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:58.354221 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.354194 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61debc59-fba7-4053-b2e5-e70feb446400-alertmanager-trusted-ca-bundle" pod "61debc59-fba7-4053-b2e5-e70feb446400" (UID: "61debc59-fba7-4053-b2e5-e70feb446400"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:56:58.354360 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.354324 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61debc59-fba7-4053-b2e5-e70feb446400-alertmanager-main-db" pod "61debc59-fba7-4053-b2e5-e70feb446400" (UID: "61debc59-fba7-4053-b2e5-e70feb446400"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:58.355179 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.355154 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-secret-alertmanager-kube-rbac-proxy-metric" pod "61debc59-fba7-4053-b2e5-e70feb446400" (UID: "61debc59-fba7-4053-b2e5-e70feb446400"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:56:58.355634 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.355591 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61debc59-fba7-4053-b2e5-e70feb446400-kube-api-access-dxjnf" pod "61debc59-fba7-4053-b2e5-e70feb446400" (UID: "61debc59-fba7-4053-b2e5-e70feb446400"). InnerVolumeSpecName "kube-api-access-dxjnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:58.357000 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.356960 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-secret-alertmanager-main-tls" pod "61debc59-fba7-4053-b2e5-e70feb446400" (UID: "61debc59-fba7-4053-b2e5-e70feb446400"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:56:58.357407 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.357370 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61debc59-fba7-4053-b2e5-e70feb446400-tls-assets" pod "61debc59-fba7-4053-b2e5-e70feb446400" (UID: "61debc59-fba7-4053-b2e5-e70feb446400"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:58.358039 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.357998 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61debc59-fba7-4053-b2e5-e70feb446400-config-out" pod "61debc59-fba7-4053-b2e5-e70feb446400" (UID: "61debc59-fba7-4053-b2e5-e70feb446400"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:58.358039 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.357969 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-secret-alertmanager-kube-rbac-proxy-web" pod "61debc59-fba7-4053-b2e5-e70feb446400" (UID: "61debc59-fba7-4053-b2e5-e70feb446400"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:56:58.359857 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.359821 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-secret-alertmanager-kube-rbac-proxy" pod "61debc59-fba7-4053-b2e5-e70feb446400" (UID: "61debc59-fba7-4053-b2e5-e70feb446400"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:56:58.360052 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.360027 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-config-volume" pod "61debc59-fba7-4053-b2e5-e70feb446400" (UID: "61debc59-fba7-4053-b2e5-e70feb446400"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:56:58.362625 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.362597 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-cluster-tls-config" pod "61debc59-fba7-4053-b2e5-e70feb446400" (UID: "61debc59-fba7-4053-b2e5-e70feb446400"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:56:58.370315 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.370293 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-web-config" pod "61debc59-fba7-4053-b2e5-e70feb446400" (UID: "61debc59-fba7-4053-b2e5-e70feb446400"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:56:58.453544 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.453515 4440 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:58.453544 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.453542 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxjnf\" (UniqueName: \"kubernetes.io/projected/61debc59-fba7-4053-b2e5-e70feb446400-kube-api-access-dxjnf\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:58.453708 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.453554 4440 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:58.453708 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.453564 4440 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-secret-alertmanager-main-tls\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:58.453708 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.453579 4440 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-cluster-tls-config\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:58.453708 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.453589 4440 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:58.453708 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.453598 4440 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/61debc59-fba7-4053-b2e5-e70feb446400-tls-assets\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:58.453708 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.453605 4440 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/61debc59-fba7-4053-b2e5-e70feb446400-config-out\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:58.453708 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.453613 4440 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61debc59-fba7-4053-b2e5-e70feb446400-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:58.453708 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.453622 4440 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-config-volume\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:58.453708 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.453630 4440 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/61debc59-fba7-4053-b2e5-e70feb446400-alertmanager-main-db\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:58.453708 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:58.453662 4440 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/61debc59-fba7-4053-b2e5-e70feb446400-web-config\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:59.066501 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.066464 4440 generic.go:356] "Generic (PLEG): container finished" podID="61debc59-fba7-4053-b2e5-e70feb446400" containerID="83f462d6cfd5698fe2cb06710d1949b684bcef5a765923d5f1a970c5e7444fbe" exitCode=0 Oct 31 00:56:59.066501 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.066490 4440 generic.go:356] "Generic (PLEG): container finished" podID="61debc59-fba7-4053-b2e5-e70feb446400" containerID="f948c1ec087dd12d18a914f0f188f509bcf4428fca8d9c640253f2e93bd622f0" exitCode=0 Oct 31 00:56:59.066501 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.066498 4440 generic.go:356] "Generic (PLEG): container finished" podID="61debc59-fba7-4053-b2e5-e70feb446400" containerID="618ffb8a6bb1004ad87b17d44abd3fd5df72a45ae7d0741ce1f0a9b6fb531bbb" exitCode=0 Oct 31 00:56:59.066501 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.066507 4440 generic.go:356] "Generic (PLEG): container finished" podID="61debc59-fba7-4053-b2e5-e70feb446400" containerID="1a729db225f690dd4323f6e84ce9a61f262c9e1c819b407ea10288ad2240600d" exitCode=0 Oct 31 00:56:59.067019 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.066514 4440 generic.go:356] "Generic (PLEG): container finished" podID="61debc59-fba7-4053-b2e5-e70feb446400" containerID="5b82a2b49bfe6f24f5cc7a611e5b5ac3f7b469d3beaa3c37832ddea136eee130" exitCode=0 Oct 31 00:56:59.067019 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.066519 4440 generic.go:356] "Generic (PLEG): container finished" podID="61debc59-fba7-4053-b2e5-e70feb446400" containerID="7e09a500742a4c1af5261cd5276860040174d50f9f990435df890960773176aa" exitCode=0 Oct 31 00:56:59.067019 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.066541 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:59.067019 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.066547 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"61debc59-fba7-4053-b2e5-e70feb446400","Type":"ContainerDied","Data":"83f462d6cfd5698fe2cb06710d1949b684bcef5a765923d5f1a970c5e7444fbe"} Oct 31 00:56:59.067019 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.066620 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"61debc59-fba7-4053-b2e5-e70feb446400","Type":"ContainerDied","Data":"f948c1ec087dd12d18a914f0f188f509bcf4428fca8d9c640253f2e93bd622f0"} Oct 31 00:56:59.067019 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.066647 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"61debc59-fba7-4053-b2e5-e70feb446400","Type":"ContainerDied","Data":"618ffb8a6bb1004ad87b17d44abd3fd5df72a45ae7d0741ce1f0a9b6fb531bbb"} Oct 31 00:56:59.067019 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.066671 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"61debc59-fba7-4053-b2e5-e70feb446400","Type":"ContainerDied","Data":"1a729db225f690dd4323f6e84ce9a61f262c9e1c819b407ea10288ad2240600d"} Oct 31 00:56:59.067019 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.066685 4440 scope.go:122] "RemoveContainer" containerID="83f462d6cfd5698fe2cb06710d1949b684bcef5a765923d5f1a970c5e7444fbe" Oct 31 00:56:59.067019 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.066696 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"61debc59-fba7-4053-b2e5-e70feb446400","Type":"ContainerDied","Data":"5b82a2b49bfe6f24f5cc7a611e5b5ac3f7b469d3beaa3c37832ddea136eee130"} Oct 31 00:56:59.067019 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.066720 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"61debc59-fba7-4053-b2e5-e70feb446400","Type":"ContainerDied","Data":"7e09a500742a4c1af5261cd5276860040174d50f9f990435df890960773176aa"} Oct 31 00:56:59.067019 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.066742 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"61debc59-fba7-4053-b2e5-e70feb446400","Type":"ContainerDied","Data":"084bb1980578b9463f2e91e69b5efbad2bbea31e9932c5a13f27ef1a6e97f9d9"} Oct 31 00:56:59.082295 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.082274 4440 scope.go:122] "RemoveContainer" containerID="f948c1ec087dd12d18a914f0f188f509bcf4428fca8d9c640253f2e93bd622f0" Oct 31 00:56:59.088907 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.088893 4440 scope.go:122] "RemoveContainer" containerID="618ffb8a6bb1004ad87b17d44abd3fd5df72a45ae7d0741ce1f0a9b6fb531bbb" Oct 31 00:56:59.099566 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.099548 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:56:59.104299 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.104279 4440 scope.go:122] "RemoveContainer" containerID="1a729db225f690dd4323f6e84ce9a61f262c9e1c819b407ea10288ad2240600d" Oct 31 00:56:59.106876 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.106858 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:56:59.110693 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.110669 4440 scope.go:122] "RemoveContainer" containerID="5b82a2b49bfe6f24f5cc7a611e5b5ac3f7b469d3beaa3c37832ddea136eee130" Oct 31 00:56:59.116770 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.116755 4440 scope.go:122] "RemoveContainer" containerID="7e09a500742a4c1af5261cd5276860040174d50f9f990435df890960773176aa" Oct 31 00:56:59.123099 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.123082 4440 scope.go:122] "RemoveContainer" containerID="a0d2691c59016a972c4acac6ab9f35e8abde11a60112c8900192ccff5710ef21" Oct 31 00:56:59.129308 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.129292 4440 scope.go:122] "RemoveContainer" containerID="83f462d6cfd5698fe2cb06710d1949b684bcef5a765923d5f1a970c5e7444fbe" Oct 31 00:56:59.129596 ip-10-0-130-39 kubenswrapper[4440]: E1031 00:56:59.129571 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83f462d6cfd5698fe2cb06710d1949b684bcef5a765923d5f1a970c5e7444fbe\": container with ID starting with 83f462d6cfd5698fe2cb06710d1949b684bcef5a765923d5f1a970c5e7444fbe not found: ID does not exist" containerID="83f462d6cfd5698fe2cb06710d1949b684bcef5a765923d5f1a970c5e7444fbe" Oct 31 00:56:59.129643 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.129603 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83f462d6cfd5698fe2cb06710d1949b684bcef5a765923d5f1a970c5e7444fbe"} err="failed to get container status \"83f462d6cfd5698fe2cb06710d1949b684bcef5a765923d5f1a970c5e7444fbe\": rpc error: code = NotFound desc = could not find container \"83f462d6cfd5698fe2cb06710d1949b684bcef5a765923d5f1a970c5e7444fbe\": container with ID starting with 83f462d6cfd5698fe2cb06710d1949b684bcef5a765923d5f1a970c5e7444fbe not found: ID does not exist" Oct 31 00:56:59.129643 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.129632 4440 scope.go:122] "RemoveContainer" containerID="f948c1ec087dd12d18a914f0f188f509bcf4428fca8d9c640253f2e93bd622f0" Oct 31 00:56:59.129846 ip-10-0-130-39 kubenswrapper[4440]: E1031 00:56:59.129831 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f948c1ec087dd12d18a914f0f188f509bcf4428fca8d9c640253f2e93bd622f0\": container with ID starting with f948c1ec087dd12d18a914f0f188f509bcf4428fca8d9c640253f2e93bd622f0 not found: ID does not exist" containerID="f948c1ec087dd12d18a914f0f188f509bcf4428fca8d9c640253f2e93bd622f0" Oct 31 00:56:59.129882 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.129850 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f948c1ec087dd12d18a914f0f188f509bcf4428fca8d9c640253f2e93bd622f0"} err="failed to get container status \"f948c1ec087dd12d18a914f0f188f509bcf4428fca8d9c640253f2e93bd622f0\": rpc error: code = NotFound desc = could not find container \"f948c1ec087dd12d18a914f0f188f509bcf4428fca8d9c640253f2e93bd622f0\": container with ID starting with f948c1ec087dd12d18a914f0f188f509bcf4428fca8d9c640253f2e93bd622f0 not found: ID does not exist" Oct 31 00:56:59.129882 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.129863 4440 scope.go:122] "RemoveContainer" containerID="618ffb8a6bb1004ad87b17d44abd3fd5df72a45ae7d0741ce1f0a9b6fb531bbb" Oct 31 00:56:59.130054 ip-10-0-130-39 kubenswrapper[4440]: E1031 00:56:59.130038 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"618ffb8a6bb1004ad87b17d44abd3fd5df72a45ae7d0741ce1f0a9b6fb531bbb\": container with ID starting with 618ffb8a6bb1004ad87b17d44abd3fd5df72a45ae7d0741ce1f0a9b6fb531bbb not found: ID does not exist" containerID="618ffb8a6bb1004ad87b17d44abd3fd5df72a45ae7d0741ce1f0a9b6fb531bbb" Oct 31 00:56:59.130097 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.130058 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"618ffb8a6bb1004ad87b17d44abd3fd5df72a45ae7d0741ce1f0a9b6fb531bbb"} err="failed to get container status \"618ffb8a6bb1004ad87b17d44abd3fd5df72a45ae7d0741ce1f0a9b6fb531bbb\": rpc error: code = NotFound desc = could not find container \"618ffb8a6bb1004ad87b17d44abd3fd5df72a45ae7d0741ce1f0a9b6fb531bbb\": container with ID starting with 618ffb8a6bb1004ad87b17d44abd3fd5df72a45ae7d0741ce1f0a9b6fb531bbb not found: ID does not exist" Oct 31 00:56:59.130097 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.130070 4440 scope.go:122] "RemoveContainer" containerID="1a729db225f690dd4323f6e84ce9a61f262c9e1c819b407ea10288ad2240600d" Oct 31 00:56:59.130263 ip-10-0-130-39 kubenswrapper[4440]: E1031 00:56:59.130248 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a729db225f690dd4323f6e84ce9a61f262c9e1c819b407ea10288ad2240600d\": container with ID starting with 1a729db225f690dd4323f6e84ce9a61f262c9e1c819b407ea10288ad2240600d not found: ID does not exist" containerID="1a729db225f690dd4323f6e84ce9a61f262c9e1c819b407ea10288ad2240600d" Oct 31 00:56:59.130302 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.130266 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a729db225f690dd4323f6e84ce9a61f262c9e1c819b407ea10288ad2240600d"} err="failed to get container status \"1a729db225f690dd4323f6e84ce9a61f262c9e1c819b407ea10288ad2240600d\": rpc error: code = NotFound desc = could not find container \"1a729db225f690dd4323f6e84ce9a61f262c9e1c819b407ea10288ad2240600d\": container with ID starting with 1a729db225f690dd4323f6e84ce9a61f262c9e1c819b407ea10288ad2240600d not found: ID does not exist" Oct 31 00:56:59.130302 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.130277 4440 scope.go:122] "RemoveContainer" containerID="5b82a2b49bfe6f24f5cc7a611e5b5ac3f7b469d3beaa3c37832ddea136eee130" Oct 31 00:56:59.130485 ip-10-0-130-39 kubenswrapper[4440]: E1031 00:56:59.130467 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b82a2b49bfe6f24f5cc7a611e5b5ac3f7b469d3beaa3c37832ddea136eee130\": container with ID starting with 5b82a2b49bfe6f24f5cc7a611e5b5ac3f7b469d3beaa3c37832ddea136eee130 not found: ID does not exist" containerID="5b82a2b49bfe6f24f5cc7a611e5b5ac3f7b469d3beaa3c37832ddea136eee130" Oct 31 00:56:59.130526 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.130488 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b82a2b49bfe6f24f5cc7a611e5b5ac3f7b469d3beaa3c37832ddea136eee130"} err="failed to get container status \"5b82a2b49bfe6f24f5cc7a611e5b5ac3f7b469d3beaa3c37832ddea136eee130\": rpc error: code = NotFound desc = could not find container \"5b82a2b49bfe6f24f5cc7a611e5b5ac3f7b469d3beaa3c37832ddea136eee130\": container with ID starting with 5b82a2b49bfe6f24f5cc7a611e5b5ac3f7b469d3beaa3c37832ddea136eee130 not found: ID does not exist" Oct 31 00:56:59.130526 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.130501 4440 scope.go:122] "RemoveContainer" containerID="7e09a500742a4c1af5261cd5276860040174d50f9f990435df890960773176aa" Oct 31 00:56:59.130712 ip-10-0-130-39 kubenswrapper[4440]: E1031 00:56:59.130691 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e09a500742a4c1af5261cd5276860040174d50f9f990435df890960773176aa\": container with ID starting with 7e09a500742a4c1af5261cd5276860040174d50f9f990435df890960773176aa not found: ID does not exist" containerID="7e09a500742a4c1af5261cd5276860040174d50f9f990435df890960773176aa" Oct 31 00:56:59.130753 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.130718 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e09a500742a4c1af5261cd5276860040174d50f9f990435df890960773176aa"} err="failed to get container status \"7e09a500742a4c1af5261cd5276860040174d50f9f990435df890960773176aa\": rpc error: code = NotFound desc = could not find container \"7e09a500742a4c1af5261cd5276860040174d50f9f990435df890960773176aa\": container with ID starting with 7e09a500742a4c1af5261cd5276860040174d50f9f990435df890960773176aa not found: ID does not exist" Oct 31 00:56:59.130753 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.130734 4440 scope.go:122] "RemoveContainer" containerID="a0d2691c59016a972c4acac6ab9f35e8abde11a60112c8900192ccff5710ef21" Oct 31 00:56:59.130959 ip-10-0-130-39 kubenswrapper[4440]: E1031 00:56:59.130942 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0d2691c59016a972c4acac6ab9f35e8abde11a60112c8900192ccff5710ef21\": container with ID starting with a0d2691c59016a972c4acac6ab9f35e8abde11a60112c8900192ccff5710ef21 not found: ID does not exist" containerID="a0d2691c59016a972c4acac6ab9f35e8abde11a60112c8900192ccff5710ef21" Oct 31 00:56:59.131002 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.130964 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0d2691c59016a972c4acac6ab9f35e8abde11a60112c8900192ccff5710ef21"} err="failed to get container status \"a0d2691c59016a972c4acac6ab9f35e8abde11a60112c8900192ccff5710ef21\": rpc error: code = NotFound desc = could not find container \"a0d2691c59016a972c4acac6ab9f35e8abde11a60112c8900192ccff5710ef21\": container with ID starting with a0d2691c59016a972c4acac6ab9f35e8abde11a60112c8900192ccff5710ef21 not found: ID does not exist" Oct 31 00:56:59.131002 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.130979 4440 scope.go:122] "RemoveContainer" containerID="83f462d6cfd5698fe2cb06710d1949b684bcef5a765923d5f1a970c5e7444fbe" Oct 31 00:56:59.131195 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.131176 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83f462d6cfd5698fe2cb06710d1949b684bcef5a765923d5f1a970c5e7444fbe"} err="failed to get container status \"83f462d6cfd5698fe2cb06710d1949b684bcef5a765923d5f1a970c5e7444fbe\": rpc error: code = NotFound desc = could not find container \"83f462d6cfd5698fe2cb06710d1949b684bcef5a765923d5f1a970c5e7444fbe\": container with ID starting with 83f462d6cfd5698fe2cb06710d1949b684bcef5a765923d5f1a970c5e7444fbe not found: ID does not exist" Oct 31 00:56:59.131239 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.131195 4440 scope.go:122] "RemoveContainer" containerID="f948c1ec087dd12d18a914f0f188f509bcf4428fca8d9c640253f2e93bd622f0" Oct 31 00:56:59.131434 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.131403 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f948c1ec087dd12d18a914f0f188f509bcf4428fca8d9c640253f2e93bd622f0"} err="failed to get container status \"f948c1ec087dd12d18a914f0f188f509bcf4428fca8d9c640253f2e93bd622f0\": rpc error: code = NotFound desc = could not find container \"f948c1ec087dd12d18a914f0f188f509bcf4428fca8d9c640253f2e93bd622f0\": container with ID starting with f948c1ec087dd12d18a914f0f188f509bcf4428fca8d9c640253f2e93bd622f0 not found: ID does not exist" Oct 31 00:56:59.131434 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.131432 4440 scope.go:122] "RemoveContainer" containerID="618ffb8a6bb1004ad87b17d44abd3fd5df72a45ae7d0741ce1f0a9b6fb531bbb" Oct 31 00:56:59.131641 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.131625 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"618ffb8a6bb1004ad87b17d44abd3fd5df72a45ae7d0741ce1f0a9b6fb531bbb"} err="failed to get container status \"618ffb8a6bb1004ad87b17d44abd3fd5df72a45ae7d0741ce1f0a9b6fb531bbb\": rpc error: code = NotFound desc = could not find container \"618ffb8a6bb1004ad87b17d44abd3fd5df72a45ae7d0741ce1f0a9b6fb531bbb\": container with ID starting with 618ffb8a6bb1004ad87b17d44abd3fd5df72a45ae7d0741ce1f0a9b6fb531bbb not found: ID does not exist" Oct 31 00:56:59.131641 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.131640 4440 scope.go:122] "RemoveContainer" containerID="1a729db225f690dd4323f6e84ce9a61f262c9e1c819b407ea10288ad2240600d" Oct 31 00:56:59.131852 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.131836 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a729db225f690dd4323f6e84ce9a61f262c9e1c819b407ea10288ad2240600d"} err="failed to get container status \"1a729db225f690dd4323f6e84ce9a61f262c9e1c819b407ea10288ad2240600d\": rpc error: code = NotFound desc = could not find container \"1a729db225f690dd4323f6e84ce9a61f262c9e1c819b407ea10288ad2240600d\": container with ID starting with 1a729db225f690dd4323f6e84ce9a61f262c9e1c819b407ea10288ad2240600d not found: ID does not exist" Oct 31 00:56:59.131852 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.131851 4440 scope.go:122] "RemoveContainer" containerID="5b82a2b49bfe6f24f5cc7a611e5b5ac3f7b469d3beaa3c37832ddea136eee130" Oct 31 00:56:59.132037 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.132022 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b82a2b49bfe6f24f5cc7a611e5b5ac3f7b469d3beaa3c37832ddea136eee130"} err="failed to get container status \"5b82a2b49bfe6f24f5cc7a611e5b5ac3f7b469d3beaa3c37832ddea136eee130\": rpc error: code = NotFound desc = could not find container \"5b82a2b49bfe6f24f5cc7a611e5b5ac3f7b469d3beaa3c37832ddea136eee130\": container with ID starting with 5b82a2b49bfe6f24f5cc7a611e5b5ac3f7b469d3beaa3c37832ddea136eee130 not found: ID does not exist" Oct 31 00:56:59.132075 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.132036 4440 scope.go:122] "RemoveContainer" containerID="7e09a500742a4c1af5261cd5276860040174d50f9f990435df890960773176aa" Oct 31 00:56:59.132221 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.132206 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e09a500742a4c1af5261cd5276860040174d50f9f990435df890960773176aa"} err="failed to get container status \"7e09a500742a4c1af5261cd5276860040174d50f9f990435df890960773176aa\": rpc error: code = NotFound desc = could not find container \"7e09a500742a4c1af5261cd5276860040174d50f9f990435df890960773176aa\": container with ID starting with 7e09a500742a4c1af5261cd5276860040174d50f9f990435df890960773176aa not found: ID does not exist" Oct 31 00:56:59.132260 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.132220 4440 scope.go:122] "RemoveContainer" containerID="a0d2691c59016a972c4acac6ab9f35e8abde11a60112c8900192ccff5710ef21" Oct 31 00:56:59.132391 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.132377 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0d2691c59016a972c4acac6ab9f35e8abde11a60112c8900192ccff5710ef21"} err="failed to get container status \"a0d2691c59016a972c4acac6ab9f35e8abde11a60112c8900192ccff5710ef21\": rpc error: code = NotFound desc = could not find container \"a0d2691c59016a972c4acac6ab9f35e8abde11a60112c8900192ccff5710ef21\": container with ID starting with a0d2691c59016a972c4acac6ab9f35e8abde11a60112c8900192ccff5710ef21 not found: ID does not exist" Oct 31 00:56:59.132391 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.132390 4440 scope.go:122] "RemoveContainer" containerID="83f462d6cfd5698fe2cb06710d1949b684bcef5a765923d5f1a970c5e7444fbe" Oct 31 00:56:59.132584 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.132570 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83f462d6cfd5698fe2cb06710d1949b684bcef5a765923d5f1a970c5e7444fbe"} err="failed to get container status \"83f462d6cfd5698fe2cb06710d1949b684bcef5a765923d5f1a970c5e7444fbe\": rpc error: code = NotFound desc = could not find container \"83f462d6cfd5698fe2cb06710d1949b684bcef5a765923d5f1a970c5e7444fbe\": container with ID starting with 83f462d6cfd5698fe2cb06710d1949b684bcef5a765923d5f1a970c5e7444fbe not found: ID does not exist" Oct 31 00:56:59.132584 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.132583 4440 scope.go:122] "RemoveContainer" containerID="f948c1ec087dd12d18a914f0f188f509bcf4428fca8d9c640253f2e93bd622f0" Oct 31 00:56:59.132759 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.132745 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f948c1ec087dd12d18a914f0f188f509bcf4428fca8d9c640253f2e93bd622f0"} err="failed to get container status \"f948c1ec087dd12d18a914f0f188f509bcf4428fca8d9c640253f2e93bd622f0\": rpc error: code = NotFound desc = could not find container \"f948c1ec087dd12d18a914f0f188f509bcf4428fca8d9c640253f2e93bd622f0\": container with ID starting with f948c1ec087dd12d18a914f0f188f509bcf4428fca8d9c640253f2e93bd622f0 not found: ID does not exist" Oct 31 00:56:59.132802 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.132759 4440 scope.go:122] "RemoveContainer" containerID="618ffb8a6bb1004ad87b17d44abd3fd5df72a45ae7d0741ce1f0a9b6fb531bbb" Oct 31 00:56:59.132930 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.132916 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"618ffb8a6bb1004ad87b17d44abd3fd5df72a45ae7d0741ce1f0a9b6fb531bbb"} err="failed to get container status \"618ffb8a6bb1004ad87b17d44abd3fd5df72a45ae7d0741ce1f0a9b6fb531bbb\": rpc error: code = NotFound desc = could not find container \"618ffb8a6bb1004ad87b17d44abd3fd5df72a45ae7d0741ce1f0a9b6fb531bbb\": container with ID starting with 618ffb8a6bb1004ad87b17d44abd3fd5df72a45ae7d0741ce1f0a9b6fb531bbb not found: ID does not exist" Oct 31 00:56:59.132970 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.132929 4440 scope.go:122] "RemoveContainer" containerID="1a729db225f690dd4323f6e84ce9a61f262c9e1c819b407ea10288ad2240600d" Oct 31 00:56:59.133107 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.133092 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a729db225f690dd4323f6e84ce9a61f262c9e1c819b407ea10288ad2240600d"} err="failed to get container status \"1a729db225f690dd4323f6e84ce9a61f262c9e1c819b407ea10288ad2240600d\": rpc error: code = NotFound desc = could not find container \"1a729db225f690dd4323f6e84ce9a61f262c9e1c819b407ea10288ad2240600d\": container with ID starting with 1a729db225f690dd4323f6e84ce9a61f262c9e1c819b407ea10288ad2240600d not found: ID does not exist" Oct 31 00:56:59.133147 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.133108 4440 scope.go:122] "RemoveContainer" containerID="5b82a2b49bfe6f24f5cc7a611e5b5ac3f7b469d3beaa3c37832ddea136eee130" Oct 31 00:56:59.133284 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.133267 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b82a2b49bfe6f24f5cc7a611e5b5ac3f7b469d3beaa3c37832ddea136eee130"} err="failed to get container status \"5b82a2b49bfe6f24f5cc7a611e5b5ac3f7b469d3beaa3c37832ddea136eee130\": rpc error: code = NotFound desc = could not find container \"5b82a2b49bfe6f24f5cc7a611e5b5ac3f7b469d3beaa3c37832ddea136eee130\": container with ID starting with 5b82a2b49bfe6f24f5cc7a611e5b5ac3f7b469d3beaa3c37832ddea136eee130 not found: ID does not exist" Oct 31 00:56:59.133330 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.133284 4440 scope.go:122] "RemoveContainer" containerID="7e09a500742a4c1af5261cd5276860040174d50f9f990435df890960773176aa" Oct 31 00:56:59.133509 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.133493 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e09a500742a4c1af5261cd5276860040174d50f9f990435df890960773176aa"} err="failed to get container status \"7e09a500742a4c1af5261cd5276860040174d50f9f990435df890960773176aa\": rpc error: code = NotFound desc = could not find container \"7e09a500742a4c1af5261cd5276860040174d50f9f990435df890960773176aa\": container with ID starting with 7e09a500742a4c1af5261cd5276860040174d50f9f990435df890960773176aa not found: ID does not exist" Oct 31 00:56:59.133509 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.133508 4440 scope.go:122] "RemoveContainer" containerID="a0d2691c59016a972c4acac6ab9f35e8abde11a60112c8900192ccff5710ef21" Oct 31 00:56:59.133703 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.133688 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0d2691c59016a972c4acac6ab9f35e8abde11a60112c8900192ccff5710ef21"} err="failed to get container status \"a0d2691c59016a972c4acac6ab9f35e8abde11a60112c8900192ccff5710ef21\": rpc error: code = NotFound desc = could not find container \"a0d2691c59016a972c4acac6ab9f35e8abde11a60112c8900192ccff5710ef21\": container with ID starting with a0d2691c59016a972c4acac6ab9f35e8abde11a60112c8900192ccff5710ef21 not found: ID does not exist" Oct 31 00:56:59.133703 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.133702 4440 scope.go:122] "RemoveContainer" containerID="83f462d6cfd5698fe2cb06710d1949b684bcef5a765923d5f1a970c5e7444fbe" Oct 31 00:56:59.133905 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.133888 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83f462d6cfd5698fe2cb06710d1949b684bcef5a765923d5f1a970c5e7444fbe"} err="failed to get container status \"83f462d6cfd5698fe2cb06710d1949b684bcef5a765923d5f1a970c5e7444fbe\": rpc error: code = NotFound desc = could not find container \"83f462d6cfd5698fe2cb06710d1949b684bcef5a765923d5f1a970c5e7444fbe\": container with ID starting with 83f462d6cfd5698fe2cb06710d1949b684bcef5a765923d5f1a970c5e7444fbe not found: ID does not exist" Oct 31 00:56:59.133905 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.133904 4440 scope.go:122] "RemoveContainer" containerID="f948c1ec087dd12d18a914f0f188f509bcf4428fca8d9c640253f2e93bd622f0" Oct 31 00:56:59.134099 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.134084 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f948c1ec087dd12d18a914f0f188f509bcf4428fca8d9c640253f2e93bd622f0"} err="failed to get container status \"f948c1ec087dd12d18a914f0f188f509bcf4428fca8d9c640253f2e93bd622f0\": rpc error: code = NotFound desc = could not find container \"f948c1ec087dd12d18a914f0f188f509bcf4428fca8d9c640253f2e93bd622f0\": container with ID starting with f948c1ec087dd12d18a914f0f188f509bcf4428fca8d9c640253f2e93bd622f0 not found: ID does not exist" Oct 31 00:56:59.134099 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.134098 4440 scope.go:122] "RemoveContainer" containerID="618ffb8a6bb1004ad87b17d44abd3fd5df72a45ae7d0741ce1f0a9b6fb531bbb" Oct 31 00:56:59.134276 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.134262 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"618ffb8a6bb1004ad87b17d44abd3fd5df72a45ae7d0741ce1f0a9b6fb531bbb"} err="failed to get container status \"618ffb8a6bb1004ad87b17d44abd3fd5df72a45ae7d0741ce1f0a9b6fb531bbb\": rpc error: code = NotFound desc = could not find container \"618ffb8a6bb1004ad87b17d44abd3fd5df72a45ae7d0741ce1f0a9b6fb531bbb\": container with ID starting with 618ffb8a6bb1004ad87b17d44abd3fd5df72a45ae7d0741ce1f0a9b6fb531bbb not found: ID does not exist" Oct 31 00:56:59.134313 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.134276 4440 scope.go:122] "RemoveContainer" containerID="1a729db225f690dd4323f6e84ce9a61f262c9e1c819b407ea10288ad2240600d" Oct 31 00:56:59.134489 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.134471 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a729db225f690dd4323f6e84ce9a61f262c9e1c819b407ea10288ad2240600d"} err="failed to get container status \"1a729db225f690dd4323f6e84ce9a61f262c9e1c819b407ea10288ad2240600d\": rpc error: code = NotFound desc = could not find container \"1a729db225f690dd4323f6e84ce9a61f262c9e1c819b407ea10288ad2240600d\": container with ID starting with 1a729db225f690dd4323f6e84ce9a61f262c9e1c819b407ea10288ad2240600d not found: ID does not exist" Oct 31 00:56:59.134541 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.134490 4440 scope.go:122] "RemoveContainer" containerID="5b82a2b49bfe6f24f5cc7a611e5b5ac3f7b469d3beaa3c37832ddea136eee130" Oct 31 00:56:59.134667 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.134652 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b82a2b49bfe6f24f5cc7a611e5b5ac3f7b469d3beaa3c37832ddea136eee130"} err="failed to get container status \"5b82a2b49bfe6f24f5cc7a611e5b5ac3f7b469d3beaa3c37832ddea136eee130\": rpc error: code = NotFound desc = could not find container \"5b82a2b49bfe6f24f5cc7a611e5b5ac3f7b469d3beaa3c37832ddea136eee130\": container with ID starting with 5b82a2b49bfe6f24f5cc7a611e5b5ac3f7b469d3beaa3c37832ddea136eee130 not found: ID does not exist" Oct 31 00:56:59.134704 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.134666 4440 scope.go:122] "RemoveContainer" containerID="7e09a500742a4c1af5261cd5276860040174d50f9f990435df890960773176aa" Oct 31 00:56:59.134868 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.134853 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e09a500742a4c1af5261cd5276860040174d50f9f990435df890960773176aa"} err="failed to get container status \"7e09a500742a4c1af5261cd5276860040174d50f9f990435df890960773176aa\": rpc error: code = NotFound desc = could not find container \"7e09a500742a4c1af5261cd5276860040174d50f9f990435df890960773176aa\": container with ID starting with 7e09a500742a4c1af5261cd5276860040174d50f9f990435df890960773176aa not found: ID does not exist" Oct 31 00:56:59.134868 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.134866 4440 scope.go:122] "RemoveContainer" containerID="a0d2691c59016a972c4acac6ab9f35e8abde11a60112c8900192ccff5710ef21" Oct 31 00:56:59.135074 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.135059 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0d2691c59016a972c4acac6ab9f35e8abde11a60112c8900192ccff5710ef21"} err="failed to get container status \"a0d2691c59016a972c4acac6ab9f35e8abde11a60112c8900192ccff5710ef21\": rpc error: code = NotFound desc = could not find container \"a0d2691c59016a972c4acac6ab9f35e8abde11a60112c8900192ccff5710ef21\": container with ID starting with a0d2691c59016a972c4acac6ab9f35e8abde11a60112c8900192ccff5710ef21 not found: ID does not exist" Oct 31 00:56:59.135074 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.135073 4440 scope.go:122] "RemoveContainer" containerID="83f462d6cfd5698fe2cb06710d1949b684bcef5a765923d5f1a970c5e7444fbe" Oct 31 00:56:59.135276 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.135260 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83f462d6cfd5698fe2cb06710d1949b684bcef5a765923d5f1a970c5e7444fbe"} err="failed to get container status \"83f462d6cfd5698fe2cb06710d1949b684bcef5a765923d5f1a970c5e7444fbe\": rpc error: code = NotFound desc = could not find container \"83f462d6cfd5698fe2cb06710d1949b684bcef5a765923d5f1a970c5e7444fbe\": container with ID starting with 83f462d6cfd5698fe2cb06710d1949b684bcef5a765923d5f1a970c5e7444fbe not found: ID does not exist" Oct 31 00:56:59.135276 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.135274 4440 scope.go:122] "RemoveContainer" containerID="f948c1ec087dd12d18a914f0f188f509bcf4428fca8d9c640253f2e93bd622f0" Oct 31 00:56:59.135478 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.135462 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f948c1ec087dd12d18a914f0f188f509bcf4428fca8d9c640253f2e93bd622f0"} err="failed to get container status \"f948c1ec087dd12d18a914f0f188f509bcf4428fca8d9c640253f2e93bd622f0\": rpc error: code = NotFound desc = could not find container \"f948c1ec087dd12d18a914f0f188f509bcf4428fca8d9c640253f2e93bd622f0\": container with ID starting with f948c1ec087dd12d18a914f0f188f509bcf4428fca8d9c640253f2e93bd622f0 not found: ID does not exist" Oct 31 00:56:59.135550 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.135478 4440 scope.go:122] "RemoveContainer" containerID="618ffb8a6bb1004ad87b17d44abd3fd5df72a45ae7d0741ce1f0a9b6fb531bbb" Oct 31 00:56:59.135707 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.135687 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"618ffb8a6bb1004ad87b17d44abd3fd5df72a45ae7d0741ce1f0a9b6fb531bbb"} err="failed to get container status \"618ffb8a6bb1004ad87b17d44abd3fd5df72a45ae7d0741ce1f0a9b6fb531bbb\": rpc error: code = NotFound desc = could not find container \"618ffb8a6bb1004ad87b17d44abd3fd5df72a45ae7d0741ce1f0a9b6fb531bbb\": container with ID starting with 618ffb8a6bb1004ad87b17d44abd3fd5df72a45ae7d0741ce1f0a9b6fb531bbb not found: ID does not exist" Oct 31 00:56:59.135749 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.135707 4440 scope.go:122] "RemoveContainer" containerID="1a729db225f690dd4323f6e84ce9a61f262c9e1c819b407ea10288ad2240600d" Oct 31 00:56:59.135903 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.135888 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a729db225f690dd4323f6e84ce9a61f262c9e1c819b407ea10288ad2240600d"} err="failed to get container status \"1a729db225f690dd4323f6e84ce9a61f262c9e1c819b407ea10288ad2240600d\": rpc error: code = NotFound desc = could not find container \"1a729db225f690dd4323f6e84ce9a61f262c9e1c819b407ea10288ad2240600d\": container with ID starting with 1a729db225f690dd4323f6e84ce9a61f262c9e1c819b407ea10288ad2240600d not found: ID does not exist" Oct 31 00:56:59.135953 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.135903 4440 scope.go:122] "RemoveContainer" containerID="5b82a2b49bfe6f24f5cc7a611e5b5ac3f7b469d3beaa3c37832ddea136eee130" Oct 31 00:56:59.136100 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.136084 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b82a2b49bfe6f24f5cc7a611e5b5ac3f7b469d3beaa3c37832ddea136eee130"} err="failed to get container status \"5b82a2b49bfe6f24f5cc7a611e5b5ac3f7b469d3beaa3c37832ddea136eee130\": rpc error: code = NotFound desc = could not find container \"5b82a2b49bfe6f24f5cc7a611e5b5ac3f7b469d3beaa3c37832ddea136eee130\": container with ID starting with 5b82a2b49bfe6f24f5cc7a611e5b5ac3f7b469d3beaa3c37832ddea136eee130 not found: ID does not exist" Oct 31 00:56:59.136145 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.136100 4440 scope.go:122] "RemoveContainer" containerID="7e09a500742a4c1af5261cd5276860040174d50f9f990435df890960773176aa" Oct 31 00:56:59.136289 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.136274 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e09a500742a4c1af5261cd5276860040174d50f9f990435df890960773176aa"} err="failed to get container status \"7e09a500742a4c1af5261cd5276860040174d50f9f990435df890960773176aa\": rpc error: code = NotFound desc = could not find container \"7e09a500742a4c1af5261cd5276860040174d50f9f990435df890960773176aa\": container with ID starting with 7e09a500742a4c1af5261cd5276860040174d50f9f990435df890960773176aa not found: ID does not exist" Oct 31 00:56:59.136329 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.136289 4440 scope.go:122] "RemoveContainer" containerID="a0d2691c59016a972c4acac6ab9f35e8abde11a60112c8900192ccff5710ef21" Oct 31 00:56:59.136497 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.136481 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0d2691c59016a972c4acac6ab9f35e8abde11a60112c8900192ccff5710ef21"} err="failed to get container status \"a0d2691c59016a972c4acac6ab9f35e8abde11a60112c8900192ccff5710ef21\": rpc error: code = NotFound desc = could not find container \"a0d2691c59016a972c4acac6ab9f35e8abde11a60112c8900192ccff5710ef21\": container with ID starting with a0d2691c59016a972c4acac6ab9f35e8abde11a60112c8900192ccff5710ef21 not found: ID does not exist" Oct 31 00:56:59.136534 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.136496 4440 scope.go:122] "RemoveContainer" containerID="83f462d6cfd5698fe2cb06710d1949b684bcef5a765923d5f1a970c5e7444fbe" Oct 31 00:56:59.136691 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.136675 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83f462d6cfd5698fe2cb06710d1949b684bcef5a765923d5f1a970c5e7444fbe"} err="failed to get container status \"83f462d6cfd5698fe2cb06710d1949b684bcef5a765923d5f1a970c5e7444fbe\": rpc error: code = NotFound desc = could not find container \"83f462d6cfd5698fe2cb06710d1949b684bcef5a765923d5f1a970c5e7444fbe\": container with ID starting with 83f462d6cfd5698fe2cb06710d1949b684bcef5a765923d5f1a970c5e7444fbe not found: ID does not exist" Oct 31 00:56:59.136733 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.136691 4440 scope.go:122] "RemoveContainer" containerID="f948c1ec087dd12d18a914f0f188f509bcf4428fca8d9c640253f2e93bd622f0" Oct 31 00:56:59.136875 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.136862 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f948c1ec087dd12d18a914f0f188f509bcf4428fca8d9c640253f2e93bd622f0"} err="failed to get container status \"f948c1ec087dd12d18a914f0f188f509bcf4428fca8d9c640253f2e93bd622f0\": rpc error: code = NotFound desc = could not find container \"f948c1ec087dd12d18a914f0f188f509bcf4428fca8d9c640253f2e93bd622f0\": container with ID starting with f948c1ec087dd12d18a914f0f188f509bcf4428fca8d9c640253f2e93bd622f0 not found: ID does not exist" Oct 31 00:56:59.136912 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.136875 4440 scope.go:122] "RemoveContainer" containerID="618ffb8a6bb1004ad87b17d44abd3fd5df72a45ae7d0741ce1f0a9b6fb531bbb" Oct 31 00:56:59.137065 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.137050 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"618ffb8a6bb1004ad87b17d44abd3fd5df72a45ae7d0741ce1f0a9b6fb531bbb"} err="failed to get container status \"618ffb8a6bb1004ad87b17d44abd3fd5df72a45ae7d0741ce1f0a9b6fb531bbb\": rpc error: code = NotFound desc = could not find container \"618ffb8a6bb1004ad87b17d44abd3fd5df72a45ae7d0741ce1f0a9b6fb531bbb\": container with ID starting with 618ffb8a6bb1004ad87b17d44abd3fd5df72a45ae7d0741ce1f0a9b6fb531bbb not found: ID does not exist" Oct 31 00:56:59.137100 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.137065 4440 scope.go:122] "RemoveContainer" containerID="1a729db225f690dd4323f6e84ce9a61f262c9e1c819b407ea10288ad2240600d" Oct 31 00:56:59.137239 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.137225 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a729db225f690dd4323f6e84ce9a61f262c9e1c819b407ea10288ad2240600d"} err="failed to get container status \"1a729db225f690dd4323f6e84ce9a61f262c9e1c819b407ea10288ad2240600d\": rpc error: code = NotFound desc = could not find container \"1a729db225f690dd4323f6e84ce9a61f262c9e1c819b407ea10288ad2240600d\": container with ID starting with 1a729db225f690dd4323f6e84ce9a61f262c9e1c819b407ea10288ad2240600d not found: ID does not exist" Oct 31 00:56:59.137279 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.137239 4440 scope.go:122] "RemoveContainer" containerID="5b82a2b49bfe6f24f5cc7a611e5b5ac3f7b469d3beaa3c37832ddea136eee130" Oct 31 00:56:59.137452 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.137436 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b82a2b49bfe6f24f5cc7a611e5b5ac3f7b469d3beaa3c37832ddea136eee130"} err="failed to get container status \"5b82a2b49bfe6f24f5cc7a611e5b5ac3f7b469d3beaa3c37832ddea136eee130\": rpc error: code = NotFound desc = could not find container \"5b82a2b49bfe6f24f5cc7a611e5b5ac3f7b469d3beaa3c37832ddea136eee130\": container with ID starting with 5b82a2b49bfe6f24f5cc7a611e5b5ac3f7b469d3beaa3c37832ddea136eee130 not found: ID does not exist" Oct 31 00:56:59.137452 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.137452 4440 scope.go:122] "RemoveContainer" containerID="7e09a500742a4c1af5261cd5276860040174d50f9f990435df890960773176aa" Oct 31 00:56:59.137659 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.137642 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e09a500742a4c1af5261cd5276860040174d50f9f990435df890960773176aa"} err="failed to get container status \"7e09a500742a4c1af5261cd5276860040174d50f9f990435df890960773176aa\": rpc error: code = NotFound desc = could not find container \"7e09a500742a4c1af5261cd5276860040174d50f9f990435df890960773176aa\": container with ID starting with 7e09a500742a4c1af5261cd5276860040174d50f9f990435df890960773176aa not found: ID does not exist" Oct 31 00:56:59.137659 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.137659 4440 scope.go:122] "RemoveContainer" containerID="a0d2691c59016a972c4acac6ab9f35e8abde11a60112c8900192ccff5710ef21" Oct 31 00:56:59.137857 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.137841 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0d2691c59016a972c4acac6ab9f35e8abde11a60112c8900192ccff5710ef21"} err="failed to get container status \"a0d2691c59016a972c4acac6ab9f35e8abde11a60112c8900192ccff5710ef21\": rpc error: code = NotFound desc = could not find container \"a0d2691c59016a972c4acac6ab9f35e8abde11a60112c8900192ccff5710ef21\": container with ID starting with a0d2691c59016a972c4acac6ab9f35e8abde11a60112c8900192ccff5710ef21 not found: ID does not exist" Oct 31 00:56:59.156710 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.156369 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:56:59.156710 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.156695 4440 cpu_manager.go:414] "containerMap: removing container" podUID="61debc59-fba7-4053-b2e5-e70feb446400" containerName="prom-label-proxy" Oct 31 00:56:59.156710 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.156711 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61debc59-fba7-4053-b2e5-e70feb446400" containerName="prom-label-proxy" Oct 31 00:56:59.156856 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.156727 4440 cpu_manager.go:414] "containerMap: removing container" podUID="61debc59-fba7-4053-b2e5-e70feb446400" containerName="kube-rbac-proxy-web" Oct 31 00:56:59.156856 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.156732 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61debc59-fba7-4053-b2e5-e70feb446400" containerName="kube-rbac-proxy-web" Oct 31 00:56:59.156856 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.156740 4440 cpu_manager.go:414] "containerMap: removing container" podUID="61debc59-fba7-4053-b2e5-e70feb446400" containerName="kube-rbac-proxy" Oct 31 00:56:59.156856 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.156744 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61debc59-fba7-4053-b2e5-e70feb446400" containerName="kube-rbac-proxy" Oct 31 00:56:59.156856 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.156754 4440 cpu_manager.go:414] "containerMap: removing container" podUID="61debc59-fba7-4053-b2e5-e70feb446400" containerName="alertmanager" Oct 31 00:56:59.156856 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.156758 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61debc59-fba7-4053-b2e5-e70feb446400" containerName="alertmanager" Oct 31 00:56:59.156856 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.156764 4440 cpu_manager.go:414] "containerMap: removing container" podUID="61debc59-fba7-4053-b2e5-e70feb446400" containerName="kube-rbac-proxy-metric" Oct 31 00:56:59.156856 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.156768 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61debc59-fba7-4053-b2e5-e70feb446400" containerName="kube-rbac-proxy-metric" Oct 31 00:56:59.156856 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.156782 4440 cpu_manager.go:414] "containerMap: removing container" podUID="61debc59-fba7-4053-b2e5-e70feb446400" containerName="config-reloader" Oct 31 00:56:59.156856 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.156787 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61debc59-fba7-4053-b2e5-e70feb446400" containerName="config-reloader" Oct 31 00:56:59.156856 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.156795 4440 cpu_manager.go:414] "containerMap: removing container" podUID="61debc59-fba7-4053-b2e5-e70feb446400" containerName="init-config-reloader" Oct 31 00:56:59.156856 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.156799 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61debc59-fba7-4053-b2e5-e70feb446400" containerName="init-config-reloader" Oct 31 00:56:59.156856 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.156846 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="61debc59-fba7-4053-b2e5-e70feb446400" containerName="config-reloader" Oct 31 00:56:59.156856 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.156854 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="61debc59-fba7-4053-b2e5-e70feb446400" containerName="prom-label-proxy" Oct 31 00:56:59.156856 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.156861 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="61debc59-fba7-4053-b2e5-e70feb446400" containerName="kube-rbac-proxy-web" Oct 31 00:56:59.157485 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.156866 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="61debc59-fba7-4053-b2e5-e70feb446400" containerName="kube-rbac-proxy" Oct 31 00:56:59.157485 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.156872 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="61debc59-fba7-4053-b2e5-e70feb446400" containerName="alertmanager" Oct 31 00:56:59.157485 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.156876 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="61debc59-fba7-4053-b2e5-e70feb446400" containerName="kube-rbac-proxy-metric" Oct 31 00:56:59.186365 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.186064 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:56:59.186365 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.186224 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:59.198964 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.198943 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 00:56:59.199199 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.199166 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 00:56:59.199199 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.199183 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-l8qqk\"" Oct 31 00:56:59.199343 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.199185 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 00:56:59.199343 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.199184 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 00:56:59.199343 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.199227 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 00:56:59.199343 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.199245 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 00:56:59.199343 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.199276 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 00:56:59.201939 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.201916 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 00:56:59.202037 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.202025 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 00:56:59.260252 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.260230 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a15e00b-582f-4f57-90cd-4bf986e2b118-config-out\") pod \"alertmanager-main-0\" (UID: \"4a15e00b-582f-4f57-90cd-4bf986e2b118\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:59.260338 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.260258 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a15e00b-582f-4f57-90cd-4bf986e2b118-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4a15e00b-582f-4f57-90cd-4bf986e2b118\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:59.260338 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.260278 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4a15e00b-582f-4f57-90cd-4bf986e2b118-config-volume\") pod \"alertmanager-main-0\" (UID: \"4a15e00b-582f-4f57-90cd-4bf986e2b118\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:59.260338 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.260298 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4a15e00b-582f-4f57-90cd-4bf986e2b118-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4a15e00b-582f-4f57-90cd-4bf986e2b118\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:59.260451 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.260359 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a15e00b-582f-4f57-90cd-4bf986e2b118-web-config\") pod \"alertmanager-main-0\" (UID: \"4a15e00b-582f-4f57-90cd-4bf986e2b118\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:59.260451 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.260394 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-572t9\" (UniqueName: \"kubernetes.io/projected/4a15e00b-582f-4f57-90cd-4bf986e2b118-kube-api-access-572t9\") pod \"alertmanager-main-0\" (UID: \"4a15e00b-582f-4f57-90cd-4bf986e2b118\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:59.260513 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.260484 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4a15e00b-582f-4f57-90cd-4bf986e2b118-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4a15e00b-582f-4f57-90cd-4bf986e2b118\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:59.260513 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.260510 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4a15e00b-582f-4f57-90cd-4bf986e2b118-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4a15e00b-582f-4f57-90cd-4bf986e2b118\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:59.260572 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.260535 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a15e00b-582f-4f57-90cd-4bf986e2b118-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4a15e00b-582f-4f57-90cd-4bf986e2b118\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:59.260605 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.260573 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a15e00b-582f-4f57-90cd-4bf986e2b118-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4a15e00b-582f-4f57-90cd-4bf986e2b118\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:59.260605 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.260591 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4a15e00b-582f-4f57-90cd-4bf986e2b118-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4a15e00b-582f-4f57-90cd-4bf986e2b118\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:59.260666 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.260609 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a15e00b-582f-4f57-90cd-4bf986e2b118-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4a15e00b-582f-4f57-90cd-4bf986e2b118\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:59.260696 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.260673 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a15e00b-582f-4f57-90cd-4bf986e2b118-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4a15e00b-582f-4f57-90cd-4bf986e2b118\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:59.360981 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.360937 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a15e00b-582f-4f57-90cd-4bf986e2b118-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4a15e00b-582f-4f57-90cd-4bf986e2b118\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:59.360981 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.360970 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a15e00b-582f-4f57-90cd-4bf986e2b118-config-out\") pod \"alertmanager-main-0\" (UID: \"4a15e00b-582f-4f57-90cd-4bf986e2b118\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:59.361112 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.360985 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a15e00b-582f-4f57-90cd-4bf986e2b118-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4a15e00b-582f-4f57-90cd-4bf986e2b118\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:59.361112 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.361005 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4a15e00b-582f-4f57-90cd-4bf986e2b118-config-volume\") pod \"alertmanager-main-0\" (UID: \"4a15e00b-582f-4f57-90cd-4bf986e2b118\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:59.361112 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.361024 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4a15e00b-582f-4f57-90cd-4bf986e2b118-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4a15e00b-582f-4f57-90cd-4bf986e2b118\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:59.361112 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.361045 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a15e00b-582f-4f57-90cd-4bf986e2b118-web-config\") pod \"alertmanager-main-0\" (UID: \"4a15e00b-582f-4f57-90cd-4bf986e2b118\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:59.361297 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.361165 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-572t9\" (UniqueName: \"kubernetes.io/projected/4a15e00b-582f-4f57-90cd-4bf986e2b118-kube-api-access-572t9\") pod \"alertmanager-main-0\" (UID: \"4a15e00b-582f-4f57-90cd-4bf986e2b118\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:59.361297 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.361216 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4a15e00b-582f-4f57-90cd-4bf986e2b118-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4a15e00b-582f-4f57-90cd-4bf986e2b118\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:59.361297 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.361245 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4a15e00b-582f-4f57-90cd-4bf986e2b118-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4a15e00b-582f-4f57-90cd-4bf986e2b118\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:59.361297 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.361271 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a15e00b-582f-4f57-90cd-4bf986e2b118-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4a15e00b-582f-4f57-90cd-4bf986e2b118\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:59.361529 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.361305 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a15e00b-582f-4f57-90cd-4bf986e2b118-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4a15e00b-582f-4f57-90cd-4bf986e2b118\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:59.361529 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.361342 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4a15e00b-582f-4f57-90cd-4bf986e2b118-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4a15e00b-582f-4f57-90cd-4bf986e2b118\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:59.361529 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.361371 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a15e00b-582f-4f57-90cd-4bf986e2b118-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4a15e00b-582f-4f57-90cd-4bf986e2b118\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:59.361910 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.361890 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a15e00b-582f-4f57-90cd-4bf986e2b118-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4a15e00b-582f-4f57-90cd-4bf986e2b118\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:59.362048 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.362028 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a15e00b-582f-4f57-90cd-4bf986e2b118-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4a15e00b-582f-4f57-90cd-4bf986e2b118\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:59.362321 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.362294 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4a15e00b-582f-4f57-90cd-4bf986e2b118-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4a15e00b-582f-4f57-90cd-4bf986e2b118\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:59.363894 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.363866 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a15e00b-582f-4f57-90cd-4bf986e2b118-config-out\") pod \"alertmanager-main-0\" (UID: \"4a15e00b-582f-4f57-90cd-4bf986e2b118\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:59.364023 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.363999 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4a15e00b-582f-4f57-90cd-4bf986e2b118-config-volume\") pod \"alertmanager-main-0\" (UID: \"4a15e00b-582f-4f57-90cd-4bf986e2b118\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:59.364091 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.364009 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4a15e00b-582f-4f57-90cd-4bf986e2b118-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4a15e00b-582f-4f57-90cd-4bf986e2b118\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:59.364327 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.364298 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a15e00b-582f-4f57-90cd-4bf986e2b118-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4a15e00b-582f-4f57-90cd-4bf986e2b118\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:59.364489 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.364447 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a15e00b-582f-4f57-90cd-4bf986e2b118-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4a15e00b-582f-4f57-90cd-4bf986e2b118\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:59.364792 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.364767 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a15e00b-582f-4f57-90cd-4bf986e2b118-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4a15e00b-582f-4f57-90cd-4bf986e2b118\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:59.364884 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.364840 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a15e00b-582f-4f57-90cd-4bf986e2b118-web-config\") pod \"alertmanager-main-0\" (UID: \"4a15e00b-582f-4f57-90cd-4bf986e2b118\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:59.364960 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.364940 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4a15e00b-582f-4f57-90cd-4bf986e2b118-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4a15e00b-582f-4f57-90cd-4bf986e2b118\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:59.365968 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.365949 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4a15e00b-582f-4f57-90cd-4bf986e2b118-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4a15e00b-582f-4f57-90cd-4bf986e2b118\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:59.370361 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.370343 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-572t9\" (UniqueName: \"kubernetes.io/projected/4a15e00b-582f-4f57-90cd-4bf986e2b118-kube-api-access-572t9\") pod \"alertmanager-main-0\" (UID: \"4a15e00b-582f-4f57-90cd-4bf986e2b118\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:59.563245 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.563209 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:56:59.767944 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.767914 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61debc59-fba7-4053-b2e5-e70feb446400" path="/var/lib/kubelet/pods/61debc59-fba7-4053-b2e5-e70feb446400/volumes" Oct 31 00:56:59.873121 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:56:59.873091 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:56:59.876000 ip-10-0-130-39 kubenswrapper[4440]: W1031 00:56:59.875970 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a15e00b_582f_4f57_90cd_4bf986e2b118.slice/crio-2c7ea5500cfd7d55a5e0a06bb5a82d7e5d669a9b0e6f26126d74eacc8f349181 WatchSource:0}: Error finding container 2c7ea5500cfd7d55a5e0a06bb5a82d7e5d669a9b0e6f26126d74eacc8f349181: Status 404 returned error can't find the container with id 2c7ea5500cfd7d55a5e0a06bb5a82d7e5d669a9b0e6f26126d74eacc8f349181 Oct 31 00:57:00.072602 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:00.072552 4440 generic.go:356] "Generic (PLEG): container finished" podID="4a15e00b-582f-4f57-90cd-4bf986e2b118" containerID="991b542bbeef05544db3e6cdf41f7eb56641fb8073c61726ff36b1379a380f91" exitCode=0 Oct 31 00:57:00.073063 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:00.072615 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a15e00b-582f-4f57-90cd-4bf986e2b118","Type":"ContainerDied","Data":"991b542bbeef05544db3e6cdf41f7eb56641fb8073c61726ff36b1379a380f91"} Oct 31 00:57:00.073063 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:00.072673 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a15e00b-582f-4f57-90cd-4bf986e2b118","Type":"ContainerStarted","Data":"2c7ea5500cfd7d55a5e0a06bb5a82d7e5d669a9b0e6f26126d74eacc8f349181"} Oct 31 00:57:00.998916 ip-10-0-130-39 kubenswrapper[4440]: E1031 00:57:00.998873 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 00:57:01.080171 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:01.080141 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a15e00b-582f-4f57-90cd-4bf986e2b118","Type":"ContainerStarted","Data":"91d1c498ef5f4668f83328cef4f05ef8a99bb341fa91077961a7e777a15e0703"} Oct 31 00:57:01.080629 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:01.080221 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a15e00b-582f-4f57-90cd-4bf986e2b118","Type":"ContainerStarted","Data":"fa06ce08f776a0b83f7bd2d310c2d179815829f1cf06bc7c8faa9ccbd3345f34"} Oct 31 00:57:01.080629 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:01.080262 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a15e00b-582f-4f57-90cd-4bf986e2b118","Type":"ContainerStarted","Data":"5f33e82fe1b4366ab9f8877002c8aee189af5d7042bd60d2e628eb97f2526578"} Oct 31 00:57:01.080629 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:01.080304 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a15e00b-582f-4f57-90cd-4bf986e2b118","Type":"ContainerStarted","Data":"177b62bae9372589b386b6f0cbaa31c648825e9fa7519c201ee72e0174b89a1f"} Oct 31 00:57:01.080629 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:01.080347 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a15e00b-582f-4f57-90cd-4bf986e2b118","Type":"ContainerStarted","Data":"128d7eb17f43f862a549dc0744348472688aa12f2b422b8a00388d1708577cff"} Oct 31 00:57:01.080629 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:01.080386 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a15e00b-582f-4f57-90cd-4bf986e2b118","Type":"ContainerStarted","Data":"a4def11f260a8debf6d786cdbbc1b0618105832753a4f4f88663599834ad34ca"} Oct 31 00:57:01.113193 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:01.113143 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.100261572 podStartE2EDuration="2.113130298s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.868726ms" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:57:01.11205141 +0000 UTC m=+65.470773725" watchObservedRunningTime="2026-10-31 00:57:01.113130298 +0000 UTC m=+65.471852613" Oct 31 00:57:03.872041 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:03.872002 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:03.991626 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:03.991590 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:04.356564 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:04.356530 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:04.577755 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:04.577715 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:06.429440 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:06.408513 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57d44c567c-fs4wp"] Oct 31 00:57:08.029300 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:08.029242 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5999b68fb8-9hrhl" podUID="9d8e4bad-7966-4577-b142-c3595de3500a" containerName="console" containerID="cri-o://9b66b09004f4a0f777c05290c9cb38409b91600ecdc32bb6fa39476e107aeee8" gracePeriod=15 Oct 31 00:57:08.251454 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:08.251399 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:08.309895 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:08.309826 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5999b68fb8-9hrhl" Oct 31 00:57:08.377810 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:08.377778 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:08.449867 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:08.449831 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d8e4bad-7966-4577-b142-c3595de3500a-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8e4bad-7966-4577-b142-c3595de3500a-service-ca\") pod \"9d8e4bad-7966-4577-b142-c3595de3500a\" (UID: \"9d8e4bad-7966-4577-b142-c3595de3500a\") " Oct 31 00:57:08.450032 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:08.449895 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d8e4bad-7966-4577-b142-c3595de3500a-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d8e4bad-7966-4577-b142-c3595de3500a-trusted-ca-bundle\") pod \"9d8e4bad-7966-4577-b142-c3595de3500a\" (UID: \"9d8e4bad-7966-4577-b142-c3595de3500a\") " Oct 31 00:57:08.450032 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:08.449936 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d8e4bad-7966-4577-b142-c3595de3500a-kube-api-access-gbcn9\" (UniqueName: \"kubernetes.io/projected/9d8e4bad-7966-4577-b142-c3595de3500a-kube-api-access-gbcn9\") pod \"9d8e4bad-7966-4577-b142-c3595de3500a\" (UID: \"9d8e4bad-7966-4577-b142-c3595de3500a\") " Oct 31 00:57:08.450032 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:08.449964 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d8e4bad-7966-4577-b142-c3595de3500a-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d8e4bad-7966-4577-b142-c3595de3500a-console-serving-cert\") pod \"9d8e4bad-7966-4577-b142-c3595de3500a\" (UID: \"9d8e4bad-7966-4577-b142-c3595de3500a\") " Oct 31 00:57:08.450032 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:08.449981 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d8e4bad-7966-4577-b142-c3595de3500a-console-config\" (UniqueName: \"kubernetes.io/configmap/9d8e4bad-7966-4577-b142-c3595de3500a-console-config\") pod \"9d8e4bad-7966-4577-b142-c3595de3500a\" (UID: \"9d8e4bad-7966-4577-b142-c3595de3500a\") " Oct 31 00:57:08.450240 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:08.450039 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d8e4bad-7966-4577-b142-c3595de3500a-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d8e4bad-7966-4577-b142-c3595de3500a-oauth-serving-cert\") pod \"9d8e4bad-7966-4577-b142-c3595de3500a\" (UID: \"9d8e4bad-7966-4577-b142-c3595de3500a\") " Oct 31 00:57:08.450240 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:08.450073 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d8e4bad-7966-4577-b142-c3595de3500a-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d8e4bad-7966-4577-b142-c3595de3500a-console-oauth-config\") pod \"9d8e4bad-7966-4577-b142-c3595de3500a\" (UID: \"9d8e4bad-7966-4577-b142-c3595de3500a\") " Oct 31 00:57:08.450240 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:08.450117 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d8e4bad-7966-4577-b142-c3595de3500a-tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d8e4bad-7966-4577-b142-c3595de3500a-tmp\") pod \"9d8e4bad-7966-4577-b142-c3595de3500a\" (UID: \"9d8e4bad-7966-4577-b142-c3595de3500a\") " Oct 31 00:57:08.450398 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:08.450315 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d8e4bad-7966-4577-b142-c3595de3500a-service-ca" pod "9d8e4bad-7966-4577-b142-c3595de3500a" (UID: "9d8e4bad-7966-4577-b142-c3595de3500a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:57:08.450499 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:08.450473 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d8e4bad-7966-4577-b142-c3595de3500a-trusted-ca-bundle" pod "9d8e4bad-7966-4577-b142-c3595de3500a" (UID: "9d8e4bad-7966-4577-b142-c3595de3500a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:57:08.450562 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:08.450510 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d8e4bad-7966-4577-b142-c3595de3500a-tmp" pod "9d8e4bad-7966-4577-b142-c3595de3500a" (UID: "9d8e4bad-7966-4577-b142-c3595de3500a"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:57:08.450716 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:08.450690 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d8e4bad-7966-4577-b142-c3595de3500a-oauth-serving-cert" pod "9d8e4bad-7966-4577-b142-c3595de3500a" (UID: "9d8e4bad-7966-4577-b142-c3595de3500a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:57:08.450781 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:08.450711 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d8e4bad-7966-4577-b142-c3595de3500a-console-config" pod "9d8e4bad-7966-4577-b142-c3595de3500a" (UID: "9d8e4bad-7966-4577-b142-c3595de3500a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:57:08.452426 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:08.452393 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d8e4bad-7966-4577-b142-c3595de3500a-kube-api-access-gbcn9" pod "9d8e4bad-7966-4577-b142-c3595de3500a" (UID: "9d8e4bad-7966-4577-b142-c3595de3500a"). InnerVolumeSpecName "kube-api-access-gbcn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:57:08.452604 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:08.452583 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d8e4bad-7966-4577-b142-c3595de3500a-console-oauth-config" pod "9d8e4bad-7966-4577-b142-c3595de3500a" (UID: "9d8e4bad-7966-4577-b142-c3595de3500a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:57:08.452708 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:08.452688 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d8e4bad-7966-4577-b142-c3595de3500a-console-serving-cert" pod "9d8e4bad-7966-4577-b142-c3595de3500a" (UID: "9d8e4bad-7966-4577-b142-c3595de3500a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:57:08.551157 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:08.551111 4440 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d8e4bad-7966-4577-b142-c3595de3500a-oauth-serving-cert\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:08.551157 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:08.551150 4440 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d8e4bad-7966-4577-b142-c3595de3500a-console-oauth-config\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:08.551157 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:08.551162 4440 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d8e4bad-7966-4577-b142-c3595de3500a-tmp\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:08.551383 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:08.551171 4440 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8e4bad-7966-4577-b142-c3595de3500a-service-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:08.551383 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:08.551181 4440 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d8e4bad-7966-4577-b142-c3595de3500a-trusted-ca-bundle\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:08.551383 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:08.551189 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbcn9\" (UniqueName: \"kubernetes.io/projected/9d8e4bad-7966-4577-b142-c3595de3500a-kube-api-access-gbcn9\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:08.551383 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:08.551199 4440 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d8e4bad-7966-4577-b142-c3595de3500a-console-serving-cert\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:08.551383 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:08.551208 4440 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d8e4bad-7966-4577-b142-c3595de3500a-console-config\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:09.108496 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:09.108460 4440 generic.go:356] "Generic (PLEG): container finished" podID="9d8e4bad-7966-4577-b142-c3595de3500a" containerID="9b66b09004f4a0f777c05290c9cb38409b91600ecdc32bb6fa39476e107aeee8" exitCode=0 Oct 31 00:57:09.108912 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:09.108507 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5999b68fb8-9hrhl" Oct 31 00:57:09.108912 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:09.108529 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5999b68fb8-9hrhl" event={"ID":"9d8e4bad-7966-4577-b142-c3595de3500a","Type":"ContainerDied","Data":"9b66b09004f4a0f777c05290c9cb38409b91600ecdc32bb6fa39476e107aeee8"} Oct 31 00:57:09.108912 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:09.108567 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5999b68fb8-9hrhl" event={"ID":"9d8e4bad-7966-4577-b142-c3595de3500a","Type":"ContainerDied","Data":"e750f64baf575b1eea72cb43889b60b59961a990dcd8cbdcb7643d27b561554d"} Oct 31 00:57:09.108912 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:09.108587 4440 scope.go:122] "RemoveContainer" containerID="9b66b09004f4a0f777c05290c9cb38409b91600ecdc32bb6fa39476e107aeee8" Oct 31 00:57:09.116674 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:09.116591 4440 scope.go:122] "RemoveContainer" containerID="9b66b09004f4a0f777c05290c9cb38409b91600ecdc32bb6fa39476e107aeee8" Oct 31 00:57:09.116921 ip-10-0-130-39 kubenswrapper[4440]: E1031 00:57:09.116895 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b66b09004f4a0f777c05290c9cb38409b91600ecdc32bb6fa39476e107aeee8\": container with ID starting with 9b66b09004f4a0f777c05290c9cb38409b91600ecdc32bb6fa39476e107aeee8 not found: ID does not exist" containerID="9b66b09004f4a0f777c05290c9cb38409b91600ecdc32bb6fa39476e107aeee8" Oct 31 00:57:09.117017 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:09.116927 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b66b09004f4a0f777c05290c9cb38409b91600ecdc32bb6fa39476e107aeee8"} err="failed to get container status \"9b66b09004f4a0f777c05290c9cb38409b91600ecdc32bb6fa39476e107aeee8\": rpc error: code = NotFound desc = could not find container \"9b66b09004f4a0f777c05290c9cb38409b91600ecdc32bb6fa39476e107aeee8\": container with ID starting with 9b66b09004f4a0f777c05290c9cb38409b91600ecdc32bb6fa39476e107aeee8 not found: ID does not exist" Oct 31 00:57:09.129464 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:09.129442 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5999b68fb8-9hrhl"] Oct 31 00:57:09.135038 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:09.135016 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5999b68fb8-9hrhl"] Oct 31 00:57:09.771291 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:09.771256 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d8e4bad-7966-4577-b142-c3595de3500a" path="/var/lib/kubelet/pods/9d8e4bad-7966-4577-b142-c3595de3500a/volumes" Oct 31 00:57:30.999757 ip-10-0-130-39 kubenswrapper[4440]: E1031 00:57:30.999717 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 00:57:31.436411 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:31.436373 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-57d44c567c-fs4wp" podUID="4bc93122-80c6-435a-9b02-c32d2da99d5d" containerName="console" containerID="cri-o://ebfabe8808791fca1fa7ec1df9fa57e760beacf1801fec8cc44523e3046a93b2" gracePeriod=15 Oct 31 00:57:31.666949 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:31.666922 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57d44c567c-fs4wp" Oct 31 00:57:31.824504 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:31.824385 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bc93122-80c6-435a-9b02-c32d2da99d5d-kube-api-access-lnrd7\" (UniqueName: \"kubernetes.io/projected/4bc93122-80c6-435a-9b02-c32d2da99d5d-kube-api-access-lnrd7\") pod \"4bc93122-80c6-435a-9b02-c32d2da99d5d\" (UID: \"4bc93122-80c6-435a-9b02-c32d2da99d5d\") " Oct 31 00:57:31.824504 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:31.824458 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bc93122-80c6-435a-9b02-c32d2da99d5d-tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bc93122-80c6-435a-9b02-c32d2da99d5d-tmp\") pod \"4bc93122-80c6-435a-9b02-c32d2da99d5d\" (UID: \"4bc93122-80c6-435a-9b02-c32d2da99d5d\") " Oct 31 00:57:31.824504 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:31.824486 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bc93122-80c6-435a-9b02-c32d2da99d5d-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4bc93122-80c6-435a-9b02-c32d2da99d5d-console-serving-cert\") pod \"4bc93122-80c6-435a-9b02-c32d2da99d5d\" (UID: \"4bc93122-80c6-435a-9b02-c32d2da99d5d\") " Oct 31 00:57:31.824787 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:31.824515 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bc93122-80c6-435a-9b02-c32d2da99d5d-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bc93122-80c6-435a-9b02-c32d2da99d5d-service-ca\") pod \"4bc93122-80c6-435a-9b02-c32d2da99d5d\" (UID: \"4bc93122-80c6-435a-9b02-c32d2da99d5d\") " Oct 31 00:57:31.824787 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:31.824558 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bc93122-80c6-435a-9b02-c32d2da99d5d-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bc93122-80c6-435a-9b02-c32d2da99d5d-trusted-ca-bundle\") pod \"4bc93122-80c6-435a-9b02-c32d2da99d5d\" (UID: \"4bc93122-80c6-435a-9b02-c32d2da99d5d\") " Oct 31 00:57:31.824787 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:31.824640 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bc93122-80c6-435a-9b02-c32d2da99d5d-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4bc93122-80c6-435a-9b02-c32d2da99d5d-console-oauth-config\") pod \"4bc93122-80c6-435a-9b02-c32d2da99d5d\" (UID: \"4bc93122-80c6-435a-9b02-c32d2da99d5d\") " Oct 31 00:57:31.824787 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:31.824672 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bc93122-80c6-435a-9b02-c32d2da99d5d-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4bc93122-80c6-435a-9b02-c32d2da99d5d-oauth-serving-cert\") pod \"4bc93122-80c6-435a-9b02-c32d2da99d5d\" (UID: \"4bc93122-80c6-435a-9b02-c32d2da99d5d\") " Oct 31 00:57:31.824787 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:31.824718 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bc93122-80c6-435a-9b02-c32d2da99d5d-console-config\" (UniqueName: \"kubernetes.io/configmap/4bc93122-80c6-435a-9b02-c32d2da99d5d-console-config\") pod \"4bc93122-80c6-435a-9b02-c32d2da99d5d\" (UID: \"4bc93122-80c6-435a-9b02-c32d2da99d5d\") " Oct 31 00:57:31.825041 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:31.824789 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bc93122-80c6-435a-9b02-c32d2da99d5d-tmp" pod "4bc93122-80c6-435a-9b02-c32d2da99d5d" (UID: "4bc93122-80c6-435a-9b02-c32d2da99d5d"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:57:31.825041 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:31.824937 4440 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bc93122-80c6-435a-9b02-c32d2da99d5d-tmp\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:31.825360 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:31.825325 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bc93122-80c6-435a-9b02-c32d2da99d5d-service-ca" pod "4bc93122-80c6-435a-9b02-c32d2da99d5d" (UID: "4bc93122-80c6-435a-9b02-c32d2da99d5d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:57:31.825360 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:31.825349 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bc93122-80c6-435a-9b02-c32d2da99d5d-oauth-serving-cert" pod "4bc93122-80c6-435a-9b02-c32d2da99d5d" (UID: "4bc93122-80c6-435a-9b02-c32d2da99d5d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:57:31.825591 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:31.825338 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bc93122-80c6-435a-9b02-c32d2da99d5d-console-config" pod "4bc93122-80c6-435a-9b02-c32d2da99d5d" (UID: "4bc93122-80c6-435a-9b02-c32d2da99d5d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:57:31.825591 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:31.825332 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bc93122-80c6-435a-9b02-c32d2da99d5d-trusted-ca-bundle" pod "4bc93122-80c6-435a-9b02-c32d2da99d5d" (UID: "4bc93122-80c6-435a-9b02-c32d2da99d5d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:57:31.826787 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:31.826758 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bc93122-80c6-435a-9b02-c32d2da99d5d-console-serving-cert" pod "4bc93122-80c6-435a-9b02-c32d2da99d5d" (UID: "4bc93122-80c6-435a-9b02-c32d2da99d5d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:57:31.826878 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:31.826836 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bc93122-80c6-435a-9b02-c32d2da99d5d-kube-api-access-lnrd7" pod "4bc93122-80c6-435a-9b02-c32d2da99d5d" (UID: "4bc93122-80c6-435a-9b02-c32d2da99d5d"). InnerVolumeSpecName "kube-api-access-lnrd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:57:31.826878 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:31.826836 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bc93122-80c6-435a-9b02-c32d2da99d5d-console-oauth-config" pod "4bc93122-80c6-435a-9b02-c32d2da99d5d" (UID: "4bc93122-80c6-435a-9b02-c32d2da99d5d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:57:31.926222 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:31.926181 4440 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4bc93122-80c6-435a-9b02-c32d2da99d5d-console-oauth-config\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:31.926222 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:31.926214 4440 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4bc93122-80c6-435a-9b02-c32d2da99d5d-oauth-serving-cert\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:31.926222 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:31.926224 4440 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4bc93122-80c6-435a-9b02-c32d2da99d5d-console-config\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:31.926487 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:31.926233 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnrd7\" (UniqueName: \"kubernetes.io/projected/4bc93122-80c6-435a-9b02-c32d2da99d5d-kube-api-access-lnrd7\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:31.926487 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:31.926244 4440 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4bc93122-80c6-435a-9b02-c32d2da99d5d-console-serving-cert\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:31.926487 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:31.926252 4440 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4bc93122-80c6-435a-9b02-c32d2da99d5d-service-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:31.926487 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:31.926260 4440 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bc93122-80c6-435a-9b02-c32d2da99d5d-trusted-ca-bundle\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:57:32.187014 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:32.186977 4440 generic.go:356] "Generic (PLEG): container finished" podID="4bc93122-80c6-435a-9b02-c32d2da99d5d" containerID="ebfabe8808791fca1fa7ec1df9fa57e760beacf1801fec8cc44523e3046a93b2" exitCode=0 Oct 31 00:57:32.187534 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:32.187027 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57d44c567c-fs4wp" Oct 31 00:57:32.187534 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:32.187054 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57d44c567c-fs4wp" event={"ID":"4bc93122-80c6-435a-9b02-c32d2da99d5d","Type":"ContainerDied","Data":"ebfabe8808791fca1fa7ec1df9fa57e760beacf1801fec8cc44523e3046a93b2"} Oct 31 00:57:32.187534 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:32.187104 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57d44c567c-fs4wp" event={"ID":"4bc93122-80c6-435a-9b02-c32d2da99d5d","Type":"ContainerDied","Data":"94d2d9dbbb5a7e2d6abb30a4127cec23b258d1f9ffd28298d0154d278c6e57d0"} Oct 31 00:57:32.187534 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:32.187133 4440 scope.go:122] "RemoveContainer" containerID="ebfabe8808791fca1fa7ec1df9fa57e760beacf1801fec8cc44523e3046a93b2" Oct 31 00:57:32.195443 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:32.195403 4440 scope.go:122] "RemoveContainer" containerID="ebfabe8808791fca1fa7ec1df9fa57e760beacf1801fec8cc44523e3046a93b2" Oct 31 00:57:32.195722 ip-10-0-130-39 kubenswrapper[4440]: E1031 00:57:32.195703 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebfabe8808791fca1fa7ec1df9fa57e760beacf1801fec8cc44523e3046a93b2\": container with ID starting with ebfabe8808791fca1fa7ec1df9fa57e760beacf1801fec8cc44523e3046a93b2 not found: ID does not exist" containerID="ebfabe8808791fca1fa7ec1df9fa57e760beacf1801fec8cc44523e3046a93b2" Oct 31 00:57:32.195786 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:32.195733 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebfabe8808791fca1fa7ec1df9fa57e760beacf1801fec8cc44523e3046a93b2"} err="failed to get container status \"ebfabe8808791fca1fa7ec1df9fa57e760beacf1801fec8cc44523e3046a93b2\": rpc error: code = NotFound desc = could not find container \"ebfabe8808791fca1fa7ec1df9fa57e760beacf1801fec8cc44523e3046a93b2\": container with ID starting with ebfabe8808791fca1fa7ec1df9fa57e760beacf1801fec8cc44523e3046a93b2 not found: ID does not exist" Oct 31 00:57:32.209471 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:32.209441 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57d44c567c-fs4wp"] Oct 31 00:57:32.214148 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:32.214123 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57d44c567c-fs4wp"] Oct 31 00:57:33.768633 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:33.768598 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bc93122-80c6-435a-9b02-c32d2da99d5d" path="/var/lib/kubelet/pods/4bc93122-80c6-435a-9b02-c32d2da99d5d/volumes" Oct 31 00:57:44.292450 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:44.292401 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:44.399829 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:44.399792 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:45.136957 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:57:45.136918 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:00.999131 ip-10-0-130-39 kubenswrapper[4440]: E1031 00:58:00.999094 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 00:58:08.851347 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:58:08.851261 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:08.958513 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:58:08.958480 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:30.999555 ip-10-0-130-39 kubenswrapper[4440]: E1031 00:58:30.999519 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 00:58:45.183199 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:58:45.183159 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:00.998391 ip-10-0-130-39 kubenswrapper[4440]: E1031 00:59:00.998356 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 00:59:30.998156 ip-10-0-130-39 kubenswrapper[4440]: E1031 00:59:30.998120 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 00:59:45.241930 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:59:45.241850 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:52.847003 ip-10-0-130-39 kubenswrapper[4440]: I1031 00:59:52.846964 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:00.999248 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:00:00.999209 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:00:24.217300 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:00:24.217255 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:24.331438 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:00:24.331384 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:27.157617 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:00:27.157574 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:27.255204 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:00:27.255168 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:30.999495 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:00:30.999460 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:00:45.297735 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:00:45.297699 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:48.867639 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:00:48.867598 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:48.975565 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:00:48.975527 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:00.998497 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:01:00.998460 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:01:30.998696 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:01:30.998618 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:01:45.373088 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:01:45.373047 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:00.998766 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:02:00.998731 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:02:09.147001 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:09.146963 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:09.274384 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:09.274341 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:30.998304 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:02:30.998267 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:02:44.856916 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:44.856829 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:45.021851 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:45.021812 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:45.130910 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:45.130834 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:45.387477 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:45.387373 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:46.309603 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:46.309566 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:46.445735 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:46.445706 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lcqfh"] Oct 31 01:02:46.445965 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:46.445955 4440 cpu_manager.go:414] "containerMap: removing container" podUID="4bc93122-80c6-435a-9b02-c32d2da99d5d" containerName="console" Oct 31 01:02:46.446006 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:46.445966 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bc93122-80c6-435a-9b02-c32d2da99d5d" containerName="console" Oct 31 01:02:46.446006 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:46.445978 4440 cpu_manager.go:414] "containerMap: removing container" podUID="9d8e4bad-7966-4577-b142-c3595de3500a" containerName="console" Oct 31 01:02:46.446006 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:46.445984 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d8e4bad-7966-4577-b142-c3595de3500a" containerName="console" Oct 31 01:02:46.446141 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:46.446023 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bc93122-80c6-435a-9b02-c32d2da99d5d" containerName="console" Oct 31 01:02:46.446141 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:46.446032 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d8e4bad-7966-4577-b142-c3595de3500a" containerName="console" Oct 31 01:02:46.448042 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:46.448020 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lcqfh" Oct 31 01:02:46.449980 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:46.449958 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:46.452908 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:46.452887 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 31 01:02:46.452998 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:46.452937 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-wfwqh\"" Oct 31 01:02:46.452998 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:46.452958 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 31 01:02:46.458273 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:46.458243 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lcqfh"] Oct 31 01:02:46.511310 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:46.511281 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/885757ee-5daa-4621-a6b8-68cf24a7b4c5-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lcqfh\" (UID: \"885757ee-5daa-4621-a6b8-68cf24a7b4c5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lcqfh" Oct 31 01:02:46.511463 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:46.511314 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/885757ee-5daa-4621-a6b8-68cf24a7b4c5-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lcqfh\" (UID: \"885757ee-5daa-4621-a6b8-68cf24a7b4c5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lcqfh" Oct 31 01:02:46.511463 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:46.511344 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x8xx\" (UniqueName: \"kubernetes.io/projected/885757ee-5daa-4621-a6b8-68cf24a7b4c5-kube-api-access-2x8xx\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lcqfh\" (UID: \"885757ee-5daa-4621-a6b8-68cf24a7b4c5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lcqfh" Oct 31 01:02:46.612254 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:46.612173 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/885757ee-5daa-4621-a6b8-68cf24a7b4c5-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lcqfh\" (UID: \"885757ee-5daa-4621-a6b8-68cf24a7b4c5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lcqfh" Oct 31 01:02:46.612254 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:46.612229 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/885757ee-5daa-4621-a6b8-68cf24a7b4c5-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lcqfh\" (UID: \"885757ee-5daa-4621-a6b8-68cf24a7b4c5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lcqfh" Oct 31 01:02:46.612410 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:46.612267 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x8xx\" (UniqueName: \"kubernetes.io/projected/885757ee-5daa-4621-a6b8-68cf24a7b4c5-kube-api-access-2x8xx\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lcqfh\" (UID: \"885757ee-5daa-4621-a6b8-68cf24a7b4c5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lcqfh" Oct 31 01:02:46.612631 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:46.612611 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/885757ee-5daa-4621-a6b8-68cf24a7b4c5-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lcqfh\" (UID: \"885757ee-5daa-4621-a6b8-68cf24a7b4c5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lcqfh" Oct 31 01:02:46.612691 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:46.612640 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/885757ee-5daa-4621-a6b8-68cf24a7b4c5-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lcqfh\" (UID: \"885757ee-5daa-4621-a6b8-68cf24a7b4c5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lcqfh" Oct 31 01:02:46.620863 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:46.620845 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x8xx\" (UniqueName: \"kubernetes.io/projected/885757ee-5daa-4621-a6b8-68cf24a7b4c5-kube-api-access-2x8xx\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lcqfh\" (UID: \"885757ee-5daa-4621-a6b8-68cf24a7b4c5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lcqfh" Oct 31 01:02:46.762486 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:46.762465 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lcqfh" Oct 31 01:02:47.034401 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:47.034362 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lcqfh"] Oct 31 01:02:47.038231 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:02:47.038205 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod885757ee_5daa_4621_a6b8_68cf24a7b4c5.slice/crio-60e7f82deaa5f955bcf970be7a32edf04cb6c20613e55be33021b904c96b9cbf WatchSource:0}: Error finding container 60e7f82deaa5f955bcf970be7a32edf04cb6c20613e55be33021b904c96b9cbf: Status 404 returned error can't find the container with id 60e7f82deaa5f955bcf970be7a32edf04cb6c20613e55be33021b904c96b9cbf Oct 31 01:02:47.040115 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:47.040098 4440 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 01:02:47.163062 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:47.163029 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lcqfh" event={"ID":"885757ee-5daa-4621-a6b8-68cf24a7b4c5","Type":"ContainerStarted","Data":"60e7f82deaa5f955bcf970be7a32edf04cb6c20613e55be33021b904c96b9cbf"} Oct 31 01:02:47.579071 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:47.579041 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:47.746064 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:47.746029 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxdnw9"] Oct 31 01:02:47.748311 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:47.748288 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxdnw9" Oct 31 01:02:47.752955 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:47.752594 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-vq275\"" Oct 31 01:02:47.762108 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:47.762088 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxdnw9"] Oct 31 01:02:47.821995 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:47.821956 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxr76\" (UniqueName: \"kubernetes.io/projected/e05eb907-6ddc-4767-9d8b-aad5f0ed41e1-kube-api-access-vxr76\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxdnw9\" (UID: \"e05eb907-6ddc-4767-9d8b-aad5f0ed41e1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxdnw9" Oct 31 01:02:47.822152 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:47.822014 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e05eb907-6ddc-4767-9d8b-aad5f0ed41e1-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxdnw9\" (UID: \"e05eb907-6ddc-4767-9d8b-aad5f0ed41e1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxdnw9" Oct 31 01:02:47.822234 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:47.822134 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e05eb907-6ddc-4767-9d8b-aad5f0ed41e1-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxdnw9\" (UID: \"e05eb907-6ddc-4767-9d8b-aad5f0ed41e1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxdnw9" Oct 31 01:02:47.838170 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:47.838104 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acwd27"] Oct 31 01:02:47.840789 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:47.840764 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acwd27" Oct 31 01:02:47.845914 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:47.845891 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-vqblg\"" Oct 31 01:02:47.851034 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:47.851009 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acwd27"] Oct 31 01:02:47.923100 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:47.923058 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e05eb907-6ddc-4767-9d8b-aad5f0ed41e1-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxdnw9\" (UID: \"e05eb907-6ddc-4767-9d8b-aad5f0ed41e1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxdnw9" Oct 31 01:02:47.923100 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:47.923103 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5ffk\" (UniqueName: \"kubernetes.io/projected/489ce419-a3c4-4de9-b2a4-4fdaff265be9-kube-api-access-k5ffk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acwd27\" (UID: \"489ce419-a3c4-4de9-b2a4-4fdaff265be9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acwd27" Oct 31 01:02:47.923325 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:47.923139 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/489ce419-a3c4-4de9-b2a4-4fdaff265be9-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acwd27\" (UID: \"489ce419-a3c4-4de9-b2a4-4fdaff265be9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acwd27" Oct 31 01:02:47.923325 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:47.923163 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxr76\" (UniqueName: \"kubernetes.io/projected/e05eb907-6ddc-4767-9d8b-aad5f0ed41e1-kube-api-access-vxr76\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxdnw9\" (UID: \"e05eb907-6ddc-4767-9d8b-aad5f0ed41e1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxdnw9" Oct 31 01:02:47.923325 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:47.923188 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/489ce419-a3c4-4de9-b2a4-4fdaff265be9-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acwd27\" (UID: \"489ce419-a3c4-4de9-b2a4-4fdaff265be9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acwd27" Oct 31 01:02:47.923325 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:47.923218 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e05eb907-6ddc-4767-9d8b-aad5f0ed41e1-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxdnw9\" (UID: \"e05eb907-6ddc-4767-9d8b-aad5f0ed41e1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxdnw9" Oct 31 01:02:47.923738 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:47.923708 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e05eb907-6ddc-4767-9d8b-aad5f0ed41e1-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxdnw9\" (UID: \"e05eb907-6ddc-4767-9d8b-aad5f0ed41e1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxdnw9" Oct 31 01:02:47.923868 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:47.923768 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e05eb907-6ddc-4767-9d8b-aad5f0ed41e1-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxdnw9\" (UID: \"e05eb907-6ddc-4767-9d8b-aad5f0ed41e1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxdnw9" Oct 31 01:02:47.935811 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:47.935783 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxr76\" (UniqueName: \"kubernetes.io/projected/e05eb907-6ddc-4767-9d8b-aad5f0ed41e1-kube-api-access-vxr76\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxdnw9\" (UID: \"e05eb907-6ddc-4767-9d8b-aad5f0ed41e1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxdnw9" Oct 31 01:02:47.944305 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:47.944281 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rfsw9"] Oct 31 01:02:47.946817 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:47.946800 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rfsw9" Oct 31 01:02:47.950877 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:47.950853 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-z9s77\"" Oct 31 01:02:47.958267 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:47.958242 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rfsw9"] Oct 31 01:02:48.023591 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.023557 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/489ce419-a3c4-4de9-b2a4-4fdaff265be9-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acwd27\" (UID: \"489ce419-a3c4-4de9-b2a4-4fdaff265be9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acwd27" Oct 31 01:02:48.023783 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.023617 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c2ea521c-7769-4b06-9511-c2a63c2e15aa-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rfsw9\" (UID: \"c2ea521c-7769-4b06-9511-c2a63c2e15aa\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rfsw9" Oct 31 01:02:48.023783 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.023649 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9bgk\" (UniqueName: \"kubernetes.io/projected/c2ea521c-7769-4b06-9511-c2a63c2e15aa-kube-api-access-v9bgk\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rfsw9\" (UID: \"c2ea521c-7769-4b06-9511-c2a63c2e15aa\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rfsw9" Oct 31 01:02:48.023783 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.023672 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5ffk\" (UniqueName: \"kubernetes.io/projected/489ce419-a3c4-4de9-b2a4-4fdaff265be9-kube-api-access-k5ffk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acwd27\" (UID: \"489ce419-a3c4-4de9-b2a4-4fdaff265be9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acwd27" Oct 31 01:02:48.023783 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.023697 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c2ea521c-7769-4b06-9511-c2a63c2e15aa-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rfsw9\" (UID: \"c2ea521c-7769-4b06-9511-c2a63c2e15aa\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rfsw9" Oct 31 01:02:48.023988 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.023845 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/489ce419-a3c4-4de9-b2a4-4fdaff265be9-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acwd27\" (UID: \"489ce419-a3c4-4de9-b2a4-4fdaff265be9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acwd27" Oct 31 01:02:48.024043 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.024010 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/489ce419-a3c4-4de9-b2a4-4fdaff265be9-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acwd27\" (UID: \"489ce419-a3c4-4de9-b2a4-4fdaff265be9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acwd27" Oct 31 01:02:48.024190 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.024170 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/489ce419-a3c4-4de9-b2a4-4fdaff265be9-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acwd27\" (UID: \"489ce419-a3c4-4de9-b2a4-4fdaff265be9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acwd27" Oct 31 01:02:48.041627 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.041600 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5ffk\" (UniqueName: \"kubernetes.io/projected/489ce419-a3c4-4de9-b2a4-4fdaff265be9-kube-api-access-k5ffk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acwd27\" (UID: \"489ce419-a3c4-4de9-b2a4-4fdaff265be9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acwd27" Oct 31 01:02:48.067529 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.067506 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxdnw9" Oct 31 01:02:48.084675 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.084648 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9zhn"] Oct 31 01:02:48.087468 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.087437 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9zhn" Oct 31 01:02:48.094732 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.094402 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-fd9dd\"" Oct 31 01:02:48.101899 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.101879 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9zhn"] Oct 31 01:02:48.125753 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.124887 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c2ea521c-7769-4b06-9511-c2a63c2e15aa-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rfsw9\" (UID: \"c2ea521c-7769-4b06-9511-c2a63c2e15aa\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rfsw9" Oct 31 01:02:48.125753 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.124945 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9bgk\" (UniqueName: \"kubernetes.io/projected/c2ea521c-7769-4b06-9511-c2a63c2e15aa-kube-api-access-v9bgk\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rfsw9\" (UID: \"c2ea521c-7769-4b06-9511-c2a63c2e15aa\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rfsw9" Oct 31 01:02:48.125753 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.124981 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c2ea521c-7769-4b06-9511-c2a63c2e15aa-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rfsw9\" (UID: \"c2ea521c-7769-4b06-9511-c2a63c2e15aa\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rfsw9" Oct 31 01:02:48.125753 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.125295 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c2ea521c-7769-4b06-9511-c2a63c2e15aa-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rfsw9\" (UID: \"c2ea521c-7769-4b06-9511-c2a63c2e15aa\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rfsw9" Oct 31 01:02:48.125753 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.125518 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c2ea521c-7769-4b06-9511-c2a63c2e15aa-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rfsw9\" (UID: \"c2ea521c-7769-4b06-9511-c2a63c2e15aa\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rfsw9" Oct 31 01:02:48.135162 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.135134 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9bgk\" (UniqueName: \"kubernetes.io/projected/c2ea521c-7769-4b06-9511-c2a63c2e15aa-kube-api-access-v9bgk\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rfsw9\" (UID: \"c2ea521c-7769-4b06-9511-c2a63c2e15aa\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rfsw9" Oct 31 01:02:48.155929 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.155907 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acwd27" Oct 31 01:02:48.225743 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.225719 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c80a8af-321f-452c-8ee6-cabf283eaf31-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9zhn\" (UID: \"7c80a8af-321f-452c-8ee6-cabf283eaf31\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9zhn" Oct 31 01:02:48.225824 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.225788 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c80a8af-321f-452c-8ee6-cabf283eaf31-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9zhn\" (UID: \"7c80a8af-321f-452c-8ee6-cabf283eaf31\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9zhn" Oct 31 01:02:48.225824 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.225812 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdjfm\" (UniqueName: \"kubernetes.io/projected/7c80a8af-321f-452c-8ee6-cabf283eaf31-kube-api-access-jdjfm\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9zhn\" (UID: \"7c80a8af-321f-452c-8ee6-cabf283eaf31\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9zhn" Oct 31 01:02:48.262683 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.262646 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rfsw9" Oct 31 01:02:48.327140 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.327047 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c80a8af-321f-452c-8ee6-cabf283eaf31-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9zhn\" (UID: \"7c80a8af-321f-452c-8ee6-cabf283eaf31\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9zhn" Oct 31 01:02:48.327240 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.327144 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c80a8af-321f-452c-8ee6-cabf283eaf31-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9zhn\" (UID: \"7c80a8af-321f-452c-8ee6-cabf283eaf31\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9zhn" Oct 31 01:02:48.327240 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.327184 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdjfm\" (UniqueName: \"kubernetes.io/projected/7c80a8af-321f-452c-8ee6-cabf283eaf31-kube-api-access-jdjfm\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9zhn\" (UID: \"7c80a8af-321f-452c-8ee6-cabf283eaf31\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9zhn" Oct 31 01:02:48.327879 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.327855 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c80a8af-321f-452c-8ee6-cabf283eaf31-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9zhn\" (UID: \"7c80a8af-321f-452c-8ee6-cabf283eaf31\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9zhn" Oct 31 01:02:48.327976 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.327895 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c80a8af-321f-452c-8ee6-cabf283eaf31-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9zhn\" (UID: \"7c80a8af-321f-452c-8ee6-cabf283eaf31\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9zhn" Oct 31 01:02:48.349864 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.349776 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdjfm\" (UniqueName: \"kubernetes.io/projected/7c80a8af-321f-452c-8ee6-cabf283eaf31-kube-api-access-jdjfm\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9zhn\" (UID: \"7c80a8af-321f-452c-8ee6-cabf283eaf31\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9zhn" Oct 31 01:02:48.367141 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.367081 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxdnw9"] Oct 31 01:02:48.371467 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:02:48.371413 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode05eb907_6ddc_4767_9d8b_aad5f0ed41e1.slice/crio-14b1f1e26820698192eb659f01dba37b1372a1c9408ae5a9311c3268e9449d02 WatchSource:0}: Error finding container 14b1f1e26820698192eb659f01dba37b1372a1c9408ae5a9311c3268e9449d02: Status 404 returned error can't find the container with id 14b1f1e26820698192eb659f01dba37b1372a1c9408ae5a9311c3268e9449d02 Oct 31 01:02:48.403219 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.403178 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9zhn" Oct 31 01:02:48.452775 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.452731 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acwd27"] Oct 31 01:02:48.541521 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.541471 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rfsw9"] Oct 31 01:02:48.545916 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:02:48.545881 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2ea521c_7769_4b06_9511_c2a63c2e15aa.slice/crio-3a357b6a0d0d34e773ebac35532ff455fcbbaec95c5a2346a7faa800e12f8500 WatchSource:0}: Error finding container 3a357b6a0d0d34e773ebac35532ff455fcbbaec95c5a2346a7faa800e12f8500: Status 404 returned error can't find the container with id 3a357b6a0d0d34e773ebac35532ff455fcbbaec95c5a2346a7faa800e12f8500 Oct 31 01:02:48.630178 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.630104 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716w4hx"] Oct 31 01:02:48.633300 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.633279 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716w4hx" Oct 31 01:02:48.637716 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.637684 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-85zhf\"" Oct 31 01:02:48.643238 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.643192 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716w4hx"] Oct 31 01:02:48.689112 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.689080 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9zhn"] Oct 31 01:02:48.693537 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:02:48.693510 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c80a8af_321f_452c_8ee6_cabf283eaf31.slice/crio-41a1234ceb0fa62f9cea433608f71fedb05b4e0c680904f414fe7ce5f22b495d WatchSource:0}: Error finding container 41a1234ceb0fa62f9cea433608f71fedb05b4e0c680904f414fe7ce5f22b495d: Status 404 returned error can't find the container with id 41a1234ceb0fa62f9cea433608f71fedb05b4e0c680904f414fe7ce5f22b495d Oct 31 01:02:48.730471 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.730412 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6193fb88-ac70-40dc-bb5d-e37b6e821959-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716w4hx\" (UID: \"6193fb88-ac70-40dc-bb5d-e37b6e821959\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716w4hx" Oct 31 01:02:48.730635 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.730542 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2799t\" (UniqueName: \"kubernetes.io/projected/6193fb88-ac70-40dc-bb5d-e37b6e821959-kube-api-access-2799t\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716w4hx\" (UID: \"6193fb88-ac70-40dc-bb5d-e37b6e821959\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716w4hx" Oct 31 01:02:48.730691 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.730663 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6193fb88-ac70-40dc-bb5d-e37b6e821959-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716w4hx\" (UID: \"6193fb88-ac70-40dc-bb5d-e37b6e821959\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716w4hx" Oct 31 01:02:48.832012 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.831958 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6193fb88-ac70-40dc-bb5d-e37b6e821959-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716w4hx\" (UID: \"6193fb88-ac70-40dc-bb5d-e37b6e821959\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716w4hx" Oct 31 01:02:48.832205 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.832045 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6193fb88-ac70-40dc-bb5d-e37b6e821959-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716w4hx\" (UID: \"6193fb88-ac70-40dc-bb5d-e37b6e821959\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716w4hx" Oct 31 01:02:48.832205 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.832115 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2799t\" (UniqueName: \"kubernetes.io/projected/6193fb88-ac70-40dc-bb5d-e37b6e821959-kube-api-access-2799t\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716w4hx\" (UID: \"6193fb88-ac70-40dc-bb5d-e37b6e821959\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716w4hx" Oct 31 01:02:48.832402 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.832379 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6193fb88-ac70-40dc-bb5d-e37b6e821959-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716w4hx\" (UID: \"6193fb88-ac70-40dc-bb5d-e37b6e821959\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716w4hx" Oct 31 01:02:48.832486 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.832412 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6193fb88-ac70-40dc-bb5d-e37b6e821959-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716w4hx\" (UID: \"6193fb88-ac70-40dc-bb5d-e37b6e821959\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716w4hx" Oct 31 01:02:48.841532 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.841500 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2799t\" (UniqueName: \"kubernetes.io/projected/6193fb88-ac70-40dc-bb5d-e37b6e821959-kube-api-access-2799t\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716w4hx\" (UID: \"6193fb88-ac70-40dc-bb5d-e37b6e821959\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716w4hx" Oct 31 01:02:48.949815 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:48.949780 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716w4hx" Oct 31 01:02:49.181305 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:49.181212 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acwd27" event={"ID":"489ce419-a3c4-4de9-b2a4-4fdaff265be9","Type":"ContainerStarted","Data":"6a573376608c60b9a7aa81fa6f77e151c8a7dad55626f1071c2372e2de7ef7ea"} Oct 31 01:02:49.182782 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:49.182711 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9zhn" event={"ID":"7c80a8af-321f-452c-8ee6-cabf283eaf31","Type":"ContainerStarted","Data":"41a1234ceb0fa62f9cea433608f71fedb05b4e0c680904f414fe7ce5f22b495d"} Oct 31 01:02:49.184451 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:49.184377 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxdnw9" event={"ID":"e05eb907-6ddc-4767-9d8b-aad5f0ed41e1","Type":"ContainerStarted","Data":"14b1f1e26820698192eb659f01dba37b1372a1c9408ae5a9311c3268e9449d02"} Oct 31 01:02:49.185900 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:49.185847 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rfsw9" event={"ID":"c2ea521c-7769-4b06-9511-c2a63c2e15aa","Type":"ContainerStarted","Data":"3a357b6a0d0d34e773ebac35532ff455fcbbaec95c5a2346a7faa800e12f8500"} Oct 31 01:02:49.232094 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:49.231992 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716w4hx"] Oct 31 01:02:50.067197 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:50.067152 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckzpsj"] Oct 31 01:02:50.070207 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:50.070180 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckzpsj" Oct 31 01:02:50.074972 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:50.074948 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-dzjdd\"" Oct 31 01:02:50.085847 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:50.085804 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckzpsj"] Oct 31 01:02:50.144729 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:50.144674 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rpgc\" (UniqueName: \"kubernetes.io/projected/bb3f2b0e-88c7-4792-85ff-02c54f46cafd-kube-api-access-8rpgc\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckzpsj\" (UID: \"bb3f2b0e-88c7-4792-85ff-02c54f46cafd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckzpsj" Oct 31 01:02:50.144924 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:50.144737 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb3f2b0e-88c7-4792-85ff-02c54f46cafd-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckzpsj\" (UID: \"bb3f2b0e-88c7-4792-85ff-02c54f46cafd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckzpsj" Oct 31 01:02:50.144924 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:50.144818 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb3f2b0e-88c7-4792-85ff-02c54f46cafd-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckzpsj\" (UID: \"bb3f2b0e-88c7-4792-85ff-02c54f46cafd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckzpsj" Oct 31 01:02:50.195282 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:50.195241 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716w4hx" event={"ID":"6193fb88-ac70-40dc-bb5d-e37b6e821959","Type":"ContainerStarted","Data":"e7bc46acc1e50fb98be113b4373e9053dc094582348cffe73364b69e8a245ea8"} Oct 31 01:02:50.246643 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:50.245822 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb3f2b0e-88c7-4792-85ff-02c54f46cafd-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckzpsj\" (UID: \"bb3f2b0e-88c7-4792-85ff-02c54f46cafd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckzpsj" Oct 31 01:02:50.246643 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:50.245881 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rpgc\" (UniqueName: \"kubernetes.io/projected/bb3f2b0e-88c7-4792-85ff-02c54f46cafd-kube-api-access-8rpgc\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckzpsj\" (UID: \"bb3f2b0e-88c7-4792-85ff-02c54f46cafd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckzpsj" Oct 31 01:02:50.246643 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:50.245906 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb3f2b0e-88c7-4792-85ff-02c54f46cafd-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckzpsj\" (UID: \"bb3f2b0e-88c7-4792-85ff-02c54f46cafd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckzpsj" Oct 31 01:02:50.246643 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:50.246320 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb3f2b0e-88c7-4792-85ff-02c54f46cafd-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckzpsj\" (UID: \"bb3f2b0e-88c7-4792-85ff-02c54f46cafd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckzpsj" Oct 31 01:02:50.246643 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:50.246329 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb3f2b0e-88c7-4792-85ff-02c54f46cafd-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckzpsj\" (UID: \"bb3f2b0e-88c7-4792-85ff-02c54f46cafd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckzpsj" Oct 31 01:02:50.255934 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:50.255898 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rpgc\" (UniqueName: \"kubernetes.io/projected/bb3f2b0e-88c7-4792-85ff-02c54f46cafd-kube-api-access-8rpgc\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckzpsj\" (UID: \"bb3f2b0e-88c7-4792-85ff-02c54f46cafd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckzpsj" Oct 31 01:02:50.396927 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:50.396647 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckzpsj" Oct 31 01:02:50.414083 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:50.413965 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:50.538552 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:50.538519 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:50.746141 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:50.746110 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckzpsj"] Oct 31 01:02:50.755194 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:02:50.755160 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb3f2b0e_88c7_4792_85ff_02c54f46cafd.slice/crio-10eceb91d2cda51367e590400a5870683120feb58477fb227502ef6a68d34787 WatchSource:0}: Error finding container 10eceb91d2cda51367e590400a5870683120feb58477fb227502ef6a68d34787: Status 404 returned error can't find the container with id 10eceb91d2cda51367e590400a5870683120feb58477fb227502ef6a68d34787 Oct 31 01:02:51.134578 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.133728 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk"] Oct 31 01:02:51.138358 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.137456 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" Oct 31 01:02:51.152930 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.152896 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 01:02:51.153087 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.153028 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 01:02:51.153153 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.153097 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 01:02:51.154283 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.154255 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk"] Oct 31 01:02:51.160790 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.160761 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-nj7zm\"" Oct 31 01:02:51.202588 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.202545 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckzpsj" event={"ID":"bb3f2b0e-88c7-4792-85ff-02c54f46cafd","Type":"ContainerStarted","Data":"10eceb91d2cda51367e590400a5870683120feb58477fb227502ef6a68d34787"} Oct 31 01:02:51.222649 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.222612 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:51.234270 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.233698 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8"] Oct 31 01:02:51.240296 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.240267 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8" Oct 31 01:02:51.255005 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.254975 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfa83ced-0142-4d9a-9706-7edb28fda974-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-6rjmk\" (UID: \"dfa83ced-0142-4d9a-9706-7edb28fda974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" Oct 31 01:02:51.255170 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.255152 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfa83ced-0142-4d9a-9706-7edb28fda974-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-6rjmk\" (UID: \"dfa83ced-0142-4d9a-9706-7edb28fda974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" Oct 31 01:02:51.255234 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.255205 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa83ced-0142-4d9a-9706-7edb28fda974-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-6rjmk\" (UID: \"dfa83ced-0142-4d9a-9706-7edb28fda974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" Oct 31 01:02:51.255803 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.255779 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfa83ced-0142-4d9a-9706-7edb28fda974-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-6rjmk\" (UID: \"dfa83ced-0142-4d9a-9706-7edb28fda974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" Oct 31 01:02:51.255900 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.255876 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfa83ced-0142-4d9a-9706-7edb28fda974-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-6rjmk\" (UID: \"dfa83ced-0142-4d9a-9706-7edb28fda974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" Oct 31 01:02:51.255964 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.255917 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfa83ced-0142-4d9a-9706-7edb28fda974-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-6rjmk\" (UID: \"dfa83ced-0142-4d9a-9706-7edb28fda974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" Oct 31 01:02:51.255964 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.255948 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfa83ced-0142-4d9a-9706-7edb28fda974-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-6rjmk\" (UID: \"dfa83ced-0142-4d9a-9706-7edb28fda974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" Oct 31 01:02:51.256068 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.255990 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa83ced-0142-4d9a-9706-7edb28fda974-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-6rjmk\" (UID: \"dfa83ced-0142-4d9a-9706-7edb28fda974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" Oct 31 01:02:51.256068 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.256018 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkgth\" (UniqueName: \"kubernetes.io/projected/dfa83ced-0142-4d9a-9706-7edb28fda974-kube-api-access-lkgth\") pod \"maas-default-gateway-openshift-default-855f9574cf-6rjmk\" (UID: \"dfa83ced-0142-4d9a-9706-7edb28fda974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" Oct 31 01:02:51.264502 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.259654 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-hjvfl\"" Oct 31 01:02:51.276226 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.275127 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8"] Oct 31 01:02:51.357090 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.357049 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfa83ced-0142-4d9a-9706-7edb28fda974-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-6rjmk\" (UID: \"dfa83ced-0142-4d9a-9706-7edb28fda974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" Oct 31 01:02:51.357289 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.357109 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bab2f8b-1710-4327-a368-132c18e7bef9-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8\" (UID: \"0bab2f8b-1710-4327-a368-132c18e7bef9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8" Oct 31 01:02:51.357289 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.357146 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfa83ced-0142-4d9a-9706-7edb28fda974-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-6rjmk\" (UID: \"dfa83ced-0142-4d9a-9706-7edb28fda974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" Oct 31 01:02:51.357289 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.357176 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bab2f8b-1710-4327-a368-132c18e7bef9-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8\" (UID: \"0bab2f8b-1710-4327-a368-132c18e7bef9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8" Oct 31 01:02:51.357289 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.357208 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bab2f8b-1710-4327-a368-132c18e7bef9-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8\" (UID: \"0bab2f8b-1710-4327-a368-132c18e7bef9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8" Oct 31 01:02:51.357289 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.357237 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bab2f8b-1710-4327-a368-132c18e7bef9-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8\" (UID: \"0bab2f8b-1710-4327-a368-132c18e7bef9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8" Oct 31 01:02:51.357289 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.357266 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bab2f8b-1710-4327-a368-132c18e7bef9-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8\" (UID: \"0bab2f8b-1710-4327-a368-132c18e7bef9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8" Oct 31 01:02:51.357632 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.357297 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfa83ced-0142-4d9a-9706-7edb28fda974-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-6rjmk\" (UID: \"dfa83ced-0142-4d9a-9706-7edb28fda974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" Oct 31 01:02:51.357632 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.357331 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfa83ced-0142-4d9a-9706-7edb28fda974-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-6rjmk\" (UID: \"dfa83ced-0142-4d9a-9706-7edb28fda974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" Oct 31 01:02:51.357632 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.357357 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bab2f8b-1710-4327-a368-132c18e7bef9-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8\" (UID: \"0bab2f8b-1710-4327-a368-132c18e7bef9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8" Oct 31 01:02:51.357632 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.357402 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa83ced-0142-4d9a-9706-7edb28fda974-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-6rjmk\" (UID: \"dfa83ced-0142-4d9a-9706-7edb28fda974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" Oct 31 01:02:51.357632 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.357438 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bab2f8b-1710-4327-a368-132c18e7bef9-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8\" (UID: \"0bab2f8b-1710-4327-a368-132c18e7bef9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8" Oct 31 01:02:51.357632 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.357464 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdggz\" (UniqueName: \"kubernetes.io/projected/0bab2f8b-1710-4327-a368-132c18e7bef9-kube-api-access-wdggz\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8\" (UID: \"0bab2f8b-1710-4327-a368-132c18e7bef9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8" Oct 31 01:02:51.357632 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.357496 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkgth\" (UniqueName: \"kubernetes.io/projected/dfa83ced-0142-4d9a-9706-7edb28fda974-kube-api-access-lkgth\") pod \"maas-default-gateway-openshift-default-855f9574cf-6rjmk\" (UID: \"dfa83ced-0142-4d9a-9706-7edb28fda974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" Oct 31 01:02:51.357632 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.357540 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfa83ced-0142-4d9a-9706-7edb28fda974-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-6rjmk\" (UID: \"dfa83ced-0142-4d9a-9706-7edb28fda974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" Oct 31 01:02:51.357632 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.357576 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bab2f8b-1710-4327-a368-132c18e7bef9-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8\" (UID: \"0bab2f8b-1710-4327-a368-132c18e7bef9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8" Oct 31 01:02:51.357632 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.357611 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfa83ced-0142-4d9a-9706-7edb28fda974-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-6rjmk\" (UID: \"dfa83ced-0142-4d9a-9706-7edb28fda974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" Oct 31 01:02:51.358052 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.357651 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa83ced-0142-4d9a-9706-7edb28fda974-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-6rjmk\" (UID: \"dfa83ced-0142-4d9a-9706-7edb28fda974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" Oct 31 01:02:51.358144 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.358122 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa83ced-0142-4d9a-9706-7edb28fda974-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-6rjmk\" (UID: \"dfa83ced-0142-4d9a-9706-7edb28fda974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" Oct 31 01:02:51.359100 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.359079 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfa83ced-0142-4d9a-9706-7edb28fda974-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-6rjmk\" (UID: \"dfa83ced-0142-4d9a-9706-7edb28fda974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" Oct 31 01:02:51.359811 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.359788 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa83ced-0142-4d9a-9706-7edb28fda974-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-6rjmk\" (UID: \"dfa83ced-0142-4d9a-9706-7edb28fda974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" Oct 31 01:02:51.360487 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.360461 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfa83ced-0142-4d9a-9706-7edb28fda974-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-6rjmk\" (UID: \"dfa83ced-0142-4d9a-9706-7edb28fda974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" Oct 31 01:02:51.360983 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.360959 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfa83ced-0142-4d9a-9706-7edb28fda974-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-6rjmk\" (UID: \"dfa83ced-0142-4d9a-9706-7edb28fda974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" Oct 31 01:02:51.362674 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.362650 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfa83ced-0142-4d9a-9706-7edb28fda974-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-6rjmk\" (UID: \"dfa83ced-0142-4d9a-9706-7edb28fda974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" Oct 31 01:02:51.363204 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.363182 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfa83ced-0142-4d9a-9706-7edb28fda974-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-6rjmk\" (UID: \"dfa83ced-0142-4d9a-9706-7edb28fda974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" Oct 31 01:02:51.372176 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.372152 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfa83ced-0142-4d9a-9706-7edb28fda974-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-6rjmk\" (UID: \"dfa83ced-0142-4d9a-9706-7edb28fda974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" Oct 31 01:02:51.372546 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.372525 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkgth\" (UniqueName: \"kubernetes.io/projected/dfa83ced-0142-4d9a-9706-7edb28fda974-kube-api-access-lkgth\") pod \"maas-default-gateway-openshift-default-855f9574cf-6rjmk\" (UID: \"dfa83ced-0142-4d9a-9706-7edb28fda974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" Oct 31 01:02:51.410124 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.410011 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:51.456165 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.456133 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" Oct 31 01:02:51.458817 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.458782 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bab2f8b-1710-4327-a368-132c18e7bef9-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8\" (UID: \"0bab2f8b-1710-4327-a368-132c18e7bef9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8" Oct 31 01:02:51.458971 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.458848 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdggz\" (UniqueName: \"kubernetes.io/projected/0bab2f8b-1710-4327-a368-132c18e7bef9-kube-api-access-wdggz\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8\" (UID: \"0bab2f8b-1710-4327-a368-132c18e7bef9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8" Oct 31 01:02:51.458971 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.458899 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bab2f8b-1710-4327-a368-132c18e7bef9-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8\" (UID: \"0bab2f8b-1710-4327-a368-132c18e7bef9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8" Oct 31 01:02:51.459076 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.458969 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bab2f8b-1710-4327-a368-132c18e7bef9-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8\" (UID: \"0bab2f8b-1710-4327-a368-132c18e7bef9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8" Oct 31 01:02:51.459076 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.458998 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bab2f8b-1710-4327-a368-132c18e7bef9-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8\" (UID: \"0bab2f8b-1710-4327-a368-132c18e7bef9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8" Oct 31 01:02:51.459076 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.459028 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bab2f8b-1710-4327-a368-132c18e7bef9-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8\" (UID: \"0bab2f8b-1710-4327-a368-132c18e7bef9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8" Oct 31 01:02:51.459076 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.459055 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bab2f8b-1710-4327-a368-132c18e7bef9-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8\" (UID: \"0bab2f8b-1710-4327-a368-132c18e7bef9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8" Oct 31 01:02:51.459272 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.459080 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bab2f8b-1710-4327-a368-132c18e7bef9-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8\" (UID: \"0bab2f8b-1710-4327-a368-132c18e7bef9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8" Oct 31 01:02:51.459272 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.459116 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bab2f8b-1710-4327-a368-132c18e7bef9-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8\" (UID: \"0bab2f8b-1710-4327-a368-132c18e7bef9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8" Oct 31 01:02:51.461868 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.461621 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bab2f8b-1710-4327-a368-132c18e7bef9-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8\" (UID: \"0bab2f8b-1710-4327-a368-132c18e7bef9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8" Oct 31 01:02:51.462687 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.462662 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bab2f8b-1710-4327-a368-132c18e7bef9-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8\" (UID: \"0bab2f8b-1710-4327-a368-132c18e7bef9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8" Oct 31 01:02:51.462852 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.462831 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bab2f8b-1710-4327-a368-132c18e7bef9-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8\" (UID: \"0bab2f8b-1710-4327-a368-132c18e7bef9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8" Oct 31 01:02:51.463026 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.462998 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bab2f8b-1710-4327-a368-132c18e7bef9-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8\" (UID: \"0bab2f8b-1710-4327-a368-132c18e7bef9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8" Oct 31 01:02:51.463510 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.463487 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bab2f8b-1710-4327-a368-132c18e7bef9-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8\" (UID: \"0bab2f8b-1710-4327-a368-132c18e7bef9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8" Oct 31 01:02:51.466273 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.466221 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bab2f8b-1710-4327-a368-132c18e7bef9-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8\" (UID: \"0bab2f8b-1710-4327-a368-132c18e7bef9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8" Oct 31 01:02:51.468736 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.468709 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bab2f8b-1710-4327-a368-132c18e7bef9-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8\" (UID: \"0bab2f8b-1710-4327-a368-132c18e7bef9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8" Oct 31 01:02:51.471171 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.471080 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdggz\" (UniqueName: \"kubernetes.io/projected/0bab2f8b-1710-4327-a368-132c18e7bef9-kube-api-access-wdggz\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8\" (UID: \"0bab2f8b-1710-4327-a368-132c18e7bef9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8" Oct 31 01:02:51.477281 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.477256 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bab2f8b-1710-4327-a368-132c18e7bef9-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8\" (UID: \"0bab2f8b-1710-4327-a368-132c18e7bef9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8" Oct 31 01:02:51.531815 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.531777 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:51.556854 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.556822 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8" Oct 31 01:02:51.774410 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.774378 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk"] Oct 31 01:02:51.779315 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:02:51.779280 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfa83ced_0142_4d9a_9706_7edb28fda974.slice/crio-1e418eb7af3fbe24b6e0a3ce8930eb35923f97a4e3fd28f7d6073bdd2c3515f2 WatchSource:0}: Error finding container 1e418eb7af3fbe24b6e0a3ce8930eb35923f97a4e3fd28f7d6073bdd2c3515f2: Status 404 returned error can't find the container with id 1e418eb7af3fbe24b6e0a3ce8930eb35923f97a4e3fd28f7d6073bdd2c3515f2 Oct 31 01:02:51.877163 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:51.877116 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8"] Oct 31 01:02:51.887889 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:02:51.887844 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bab2f8b_1710_4327_a368_132c18e7bef9.slice/crio-734ccc0a56206681c8d3199642bac24e68b3c2e0741cae7134c97d9d677e4cbc WatchSource:0}: Error finding container 734ccc0a56206681c8d3199642bac24e68b3c2e0741cae7134c97d9d677e4cbc: Status 404 returned error can't find the container with id 734ccc0a56206681c8d3199642bac24e68b3c2e0741cae7134c97d9d677e4cbc Oct 31 01:02:52.213258 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:52.213164 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" event={"ID":"dfa83ced-0142-4d9a-9706-7edb28fda974","Type":"ContainerStarted","Data":"1e418eb7af3fbe24b6e0a3ce8930eb35923f97a4e3fd28f7d6073bdd2c3515f2"} Oct 31 01:02:52.217615 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:52.217521 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8" event={"ID":"0bab2f8b-1710-4327-a368-132c18e7bef9","Type":"ContainerStarted","Data":"734ccc0a56206681c8d3199642bac24e68b3c2e0741cae7134c97d9d677e4cbc"} Oct 31 01:02:52.622729 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:52.622646 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wb9gd"] Oct 31 01:02:52.626517 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:52.626488 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wb9gd" Oct 31 01:02:52.649438 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:52.649128 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wb9gd"] Oct 31 01:02:52.776662 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:52.776623 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3da0ab7f-932f-4367-a609-fe25d23cac66-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wb9gd\" (UID: \"3da0ab7f-932f-4367-a609-fe25d23cac66\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wb9gd" Oct 31 01:02:52.776837 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:52.776686 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3da0ab7f-932f-4367-a609-fe25d23cac66-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wb9gd\" (UID: \"3da0ab7f-932f-4367-a609-fe25d23cac66\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wb9gd" Oct 31 01:02:52.776837 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:52.776706 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp2qm\" (UniqueName: \"kubernetes.io/projected/3da0ab7f-932f-4367-a609-fe25d23cac66-kube-api-access-sp2qm\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wb9gd\" (UID: \"3da0ab7f-932f-4367-a609-fe25d23cac66\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wb9gd" Oct 31 01:02:52.877453 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:52.877353 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3da0ab7f-932f-4367-a609-fe25d23cac66-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wb9gd\" (UID: \"3da0ab7f-932f-4367-a609-fe25d23cac66\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wb9gd" Oct 31 01:02:52.877637 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:52.877504 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3da0ab7f-932f-4367-a609-fe25d23cac66-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wb9gd\" (UID: \"3da0ab7f-932f-4367-a609-fe25d23cac66\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wb9gd" Oct 31 01:02:52.877637 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:52.877569 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp2qm\" (UniqueName: \"kubernetes.io/projected/3da0ab7f-932f-4367-a609-fe25d23cac66-kube-api-access-sp2qm\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wb9gd\" (UID: \"3da0ab7f-932f-4367-a609-fe25d23cac66\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wb9gd" Oct 31 01:02:52.878403 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:52.878378 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3da0ab7f-932f-4367-a609-fe25d23cac66-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wb9gd\" (UID: \"3da0ab7f-932f-4367-a609-fe25d23cac66\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wb9gd" Oct 31 01:02:52.878872 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:52.878849 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3da0ab7f-932f-4367-a609-fe25d23cac66-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wb9gd\" (UID: \"3da0ab7f-932f-4367-a609-fe25d23cac66\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wb9gd" Oct 31 01:02:52.905198 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:52.905157 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp2qm\" (UniqueName: \"kubernetes.io/projected/3da0ab7f-932f-4367-a609-fe25d23cac66-kube-api-access-sp2qm\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wb9gd\" (UID: \"3da0ab7f-932f-4367-a609-fe25d23cac66\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wb9gd" Oct 31 01:02:52.963455 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:52.961162 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wb9gd" Oct 31 01:02:53.297217 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:53.297179 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wb9gd"] Oct 31 01:02:53.301538 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:02:53.301503 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3da0ab7f_932f_4367_a609_fe25d23cac66.slice/crio-9ab17cfc4658a2cfb9d9f9b730f8c1a1c4f918da589aeae241f07cae93644cd9 WatchSource:0}: Error finding container 9ab17cfc4658a2cfb9d9f9b730f8c1a1c4f918da589aeae241f07cae93644cd9: Status 404 returned error can't find the container with id 9ab17cfc4658a2cfb9d9f9b730f8c1a1c4f918da589aeae241f07cae93644cd9 Oct 31 01:02:54.236511 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:54.236464 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:54.257823 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:54.257783 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wb9gd" event={"ID":"3da0ab7f-932f-4367-a609-fe25d23cac66","Type":"ContainerStarted","Data":"9ab17cfc4658a2cfb9d9f9b730f8c1a1c4f918da589aeae241f07cae93644cd9"} Oct 31 01:02:55.332628 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.330650 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzqfq"] Oct 31 01:02:55.336885 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.333849 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzqfq" Oct 31 01:02:55.348216 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.348131 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzqfq"] Oct 31 01:02:55.370517 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.368563 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xltjz"] Oct 31 01:02:55.372705 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.372678 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xltjz" Oct 31 01:02:55.387818 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.387776 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xltjz"] Oct 31 01:02:55.401789 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.401754 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axh5h6"] Oct 31 01:02:55.409260 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.409227 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axh5h6" Oct 31 01:02:55.422788 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.422475 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axh5h6"] Oct 31 01:02:55.437345 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.437280 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qjfwr"] Oct 31 01:02:55.440851 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.440824 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qjfwr" Oct 31 01:02:55.449901 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.449872 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qjfwr"] Oct 31 01:02:55.511633 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.511175 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38588f65-5d75-4b58-a7f6-912cc02c2064-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xltjz\" (UID: \"38588f65-5d75-4b58-a7f6-912cc02c2064\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xltjz" Oct 31 01:02:55.511633 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.511219 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5148e11-ffda-4e2f-be35-f17c13e0515b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axh5h6\" (UID: \"d5148e11-ffda-4e2f-be35-f17c13e0515b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axh5h6" Oct 31 01:02:55.511633 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.511268 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq7xr\" (UniqueName: \"kubernetes.io/projected/d5148e11-ffda-4e2f-be35-f17c13e0515b-kube-api-access-qq7xr\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axh5h6\" (UID: \"d5148e11-ffda-4e2f-be35-f17c13e0515b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axh5h6" Oct 31 01:02:55.511633 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.511295 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38588f65-5d75-4b58-a7f6-912cc02c2064-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xltjz\" (UID: \"38588f65-5d75-4b58-a7f6-912cc02c2064\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xltjz" Oct 31 01:02:55.511633 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.511318 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5148e11-ffda-4e2f-be35-f17c13e0515b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axh5h6\" (UID: \"d5148e11-ffda-4e2f-be35-f17c13e0515b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axh5h6" Oct 31 01:02:55.511633 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.511335 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtbfv\" (UniqueName: \"kubernetes.io/projected/38588f65-5d75-4b58-a7f6-912cc02c2064-kube-api-access-rtbfv\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xltjz\" (UID: \"38588f65-5d75-4b58-a7f6-912cc02c2064\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xltjz" Oct 31 01:02:55.511633 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.511362 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9eb24e27-6ef1-44f1-bde2-130f56377c8c-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzqfq\" (UID: \"9eb24e27-6ef1-44f1-bde2-130f56377c8c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzqfq" Oct 31 01:02:55.511633 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.511382 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9eb24e27-6ef1-44f1-bde2-130f56377c8c-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzqfq\" (UID: \"9eb24e27-6ef1-44f1-bde2-130f56377c8c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzqfq" Oct 31 01:02:55.511633 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.511407 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg7sp\" (UniqueName: \"kubernetes.io/projected/9eb24e27-6ef1-44f1-bde2-130f56377c8c-kube-api-access-kg7sp\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzqfq\" (UID: \"9eb24e27-6ef1-44f1-bde2-130f56377c8c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzqfq" Oct 31 01:02:55.613978 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.612211 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq7xr\" (UniqueName: \"kubernetes.io/projected/d5148e11-ffda-4e2f-be35-f17c13e0515b-kube-api-access-qq7xr\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axh5h6\" (UID: \"d5148e11-ffda-4e2f-be35-f17c13e0515b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axh5h6" Oct 31 01:02:55.613978 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.612249 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38588f65-5d75-4b58-a7f6-912cc02c2064-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xltjz\" (UID: \"38588f65-5d75-4b58-a7f6-912cc02c2064\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xltjz" Oct 31 01:02:55.613978 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.612274 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5148e11-ffda-4e2f-be35-f17c13e0515b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axh5h6\" (UID: \"d5148e11-ffda-4e2f-be35-f17c13e0515b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axh5h6" Oct 31 01:02:55.613978 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.612297 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtbfv\" (UniqueName: \"kubernetes.io/projected/38588f65-5d75-4b58-a7f6-912cc02c2064-kube-api-access-rtbfv\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xltjz\" (UID: \"38588f65-5d75-4b58-a7f6-912cc02c2064\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xltjz" Oct 31 01:02:55.613978 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.612322 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9eb24e27-6ef1-44f1-bde2-130f56377c8c-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzqfq\" (UID: \"9eb24e27-6ef1-44f1-bde2-130f56377c8c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzqfq" Oct 31 01:02:55.613978 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.612343 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fffeb29e-3429-4a1e-a589-11367e62c490-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qjfwr\" (UID: \"fffeb29e-3429-4a1e-a589-11367e62c490\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qjfwr" Oct 31 01:02:55.613978 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.612395 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9eb24e27-6ef1-44f1-bde2-130f56377c8c-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzqfq\" (UID: \"9eb24e27-6ef1-44f1-bde2-130f56377c8c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzqfq" Oct 31 01:02:55.613978 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.612456 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftl6b\" (UniqueName: \"kubernetes.io/projected/fffeb29e-3429-4a1e-a589-11367e62c490-kube-api-access-ftl6b\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qjfwr\" (UID: \"fffeb29e-3429-4a1e-a589-11367e62c490\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qjfwr" Oct 31 01:02:55.613978 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.612493 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg7sp\" (UniqueName: \"kubernetes.io/projected/9eb24e27-6ef1-44f1-bde2-130f56377c8c-kube-api-access-kg7sp\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzqfq\" (UID: \"9eb24e27-6ef1-44f1-bde2-130f56377c8c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzqfq" Oct 31 01:02:55.613978 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.612529 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fffeb29e-3429-4a1e-a589-11367e62c490-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qjfwr\" (UID: \"fffeb29e-3429-4a1e-a589-11367e62c490\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qjfwr" Oct 31 01:02:55.613978 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.612581 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38588f65-5d75-4b58-a7f6-912cc02c2064-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xltjz\" (UID: \"38588f65-5d75-4b58-a7f6-912cc02c2064\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xltjz" Oct 31 01:02:55.613978 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.612625 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5148e11-ffda-4e2f-be35-f17c13e0515b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axh5h6\" (UID: \"d5148e11-ffda-4e2f-be35-f17c13e0515b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axh5h6" Oct 31 01:02:55.613978 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.613035 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5148e11-ffda-4e2f-be35-f17c13e0515b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axh5h6\" (UID: \"d5148e11-ffda-4e2f-be35-f17c13e0515b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axh5h6" Oct 31 01:02:55.613978 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.613381 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38588f65-5d75-4b58-a7f6-912cc02c2064-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xltjz\" (UID: \"38588f65-5d75-4b58-a7f6-912cc02c2064\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xltjz" Oct 31 01:02:55.613978 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.613398 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9eb24e27-6ef1-44f1-bde2-130f56377c8c-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzqfq\" (UID: \"9eb24e27-6ef1-44f1-bde2-130f56377c8c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzqfq" Oct 31 01:02:55.613978 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.613655 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9eb24e27-6ef1-44f1-bde2-130f56377c8c-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzqfq\" (UID: \"9eb24e27-6ef1-44f1-bde2-130f56377c8c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzqfq" Oct 31 01:02:55.613978 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.613655 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5148e11-ffda-4e2f-be35-f17c13e0515b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axh5h6\" (UID: \"d5148e11-ffda-4e2f-be35-f17c13e0515b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axh5h6" Oct 31 01:02:55.613978 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.613896 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38588f65-5d75-4b58-a7f6-912cc02c2064-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xltjz\" (UID: \"38588f65-5d75-4b58-a7f6-912cc02c2064\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xltjz" Oct 31 01:02:55.636346 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.636263 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg7sp\" (UniqueName: \"kubernetes.io/projected/9eb24e27-6ef1-44f1-bde2-130f56377c8c-kube-api-access-kg7sp\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzqfq\" (UID: \"9eb24e27-6ef1-44f1-bde2-130f56377c8c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzqfq" Oct 31 01:02:55.640036 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.640004 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq7xr\" (UniqueName: \"kubernetes.io/projected/d5148e11-ffda-4e2f-be35-f17c13e0515b-kube-api-access-qq7xr\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axh5h6\" (UID: \"d5148e11-ffda-4e2f-be35-f17c13e0515b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axh5h6" Oct 31 01:02:55.653896 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.653626 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzqfq" Oct 31 01:02:55.655460 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.655327 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtbfv\" (UniqueName: \"kubernetes.io/projected/38588f65-5d75-4b58-a7f6-912cc02c2064-kube-api-access-rtbfv\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xltjz\" (UID: \"38588f65-5d75-4b58-a7f6-912cc02c2064\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xltjz" Oct 31 01:02:55.699536 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.699307 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xltjz" Oct 31 01:02:55.717309 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.716080 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftl6b\" (UniqueName: \"kubernetes.io/projected/fffeb29e-3429-4a1e-a589-11367e62c490-kube-api-access-ftl6b\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qjfwr\" (UID: \"fffeb29e-3429-4a1e-a589-11367e62c490\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qjfwr" Oct 31 01:02:55.717309 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.716143 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fffeb29e-3429-4a1e-a589-11367e62c490-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qjfwr\" (UID: \"fffeb29e-3429-4a1e-a589-11367e62c490\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qjfwr" Oct 31 01:02:55.717309 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.716252 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fffeb29e-3429-4a1e-a589-11367e62c490-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qjfwr\" (UID: \"fffeb29e-3429-4a1e-a589-11367e62c490\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qjfwr" Oct 31 01:02:55.717309 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.716648 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fffeb29e-3429-4a1e-a589-11367e62c490-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qjfwr\" (UID: \"fffeb29e-3429-4a1e-a589-11367e62c490\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qjfwr" Oct 31 01:02:55.717309 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.717106 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fffeb29e-3429-4a1e-a589-11367e62c490-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qjfwr\" (UID: \"fffeb29e-3429-4a1e-a589-11367e62c490\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qjfwr" Oct 31 01:02:55.732841 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.732725 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axh5h6" Oct 31 01:02:55.737446 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.737402 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftl6b\" (UniqueName: \"kubernetes.io/projected/fffeb29e-3429-4a1e-a589-11367e62c490-kube-api-access-ftl6b\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qjfwr\" (UID: \"fffeb29e-3429-4a1e-a589-11367e62c490\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qjfwr" Oct 31 01:02:55.768020 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.765032 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qjfwr" Oct 31 01:02:55.995917 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:55.995817 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzqfq"] Oct 31 01:02:56.012152 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:02:56.012111 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eb24e27_6ef1_44f1_bde2_130f56377c8c.slice/crio-a17ccf8cab2705b533a8bbb74b9d5a189fd4b10e85b32ea96d8a29515e4a8992 WatchSource:0}: Error finding container a17ccf8cab2705b533a8bbb74b9d5a189fd4b10e85b32ea96d8a29515e4a8992: Status 404 returned error can't find the container with id a17ccf8cab2705b533a8bbb74b9d5a189fd4b10e85b32ea96d8a29515e4a8992 Oct 31 01:02:56.072605 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:56.071813 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xltjz"] Oct 31 01:02:56.103160 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:56.103134 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axh5h6"] Oct 31 01:02:56.109618 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:02:56.109583 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5148e11_ffda_4e2f_be35_f17c13e0515b.slice/crio-379b27a0018ea59d9bdb7324efeb1bc8adb94f6d9b9e7824e9b596177952db9c WatchSource:0}: Error finding container 379b27a0018ea59d9bdb7324efeb1bc8adb94f6d9b9e7824e9b596177952db9c: Status 404 returned error can't find the container with id 379b27a0018ea59d9bdb7324efeb1bc8adb94f6d9b9e7824e9b596177952db9c Oct 31 01:02:56.148687 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:56.148606 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qjfwr"] Oct 31 01:02:56.306766 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:56.306673 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axh5h6" event={"ID":"d5148e11-ffda-4e2f-be35-f17c13e0515b","Type":"ContainerStarted","Data":"379b27a0018ea59d9bdb7324efeb1bc8adb94f6d9b9e7824e9b596177952db9c"} Oct 31 01:02:56.316615 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:56.315673 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qjfwr" event={"ID":"fffeb29e-3429-4a1e-a589-11367e62c490","Type":"ContainerStarted","Data":"db9533e268c4680ffd29272c63ab6c77949f19e7f105c032c156ec1a29e57fda"} Oct 31 01:02:56.329060 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:56.328929 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xltjz" event={"ID":"38588f65-5d75-4b58-a7f6-912cc02c2064","Type":"ContainerStarted","Data":"bd0eba94547e3d1cf59f2432b41ffc1b68ca74142ef6a5e5596a636e4e3fcbbd"} Oct 31 01:02:56.340184 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:02:56.340099 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzqfq" event={"ID":"9eb24e27-6ef1-44f1-bde2-130f56377c8c","Type":"ContainerStarted","Data":"a17ccf8cab2705b533a8bbb74b9d5a189fd4b10e85b32ea96d8a29515e4a8992"} Oct 31 01:03:01.029221 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:03:01.029099 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:03:04.346089 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:04.345503 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:03:04.346089 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:04.345600 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:03:04.346089 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:04.345646 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:03:04.368060 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:04.367289 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:03:04.368060 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:04.367380 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:03:04.368060 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:04.367441 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:03:05.529275 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:05.529226 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" event={"ID":"dfa83ced-0142-4d9a-9706-7edb28fda974","Type":"ContainerStarted","Data":"d0284841f74e813cc44cba636b2a11fcecb5e3a84770afb2bfc7b37cbfa6ed5e"} Oct 31 01:03:05.539456 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:05.539097 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8" event={"ID":"0bab2f8b-1710-4327-a368-132c18e7bef9","Type":"ContainerStarted","Data":"e5d27b0aa94d9e1e719917412595d706d8d341a94e77cd4b12d64a8b7df0d850"} Oct 31 01:03:05.560677 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:05.560595 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8" Oct 31 01:03:05.566649 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:05.566557 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8" Oct 31 01:03:05.580987 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:05.580573 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" podStartSLOduration=2.001704486 podStartE2EDuration="14.580555443s" totalImagesPullingTime="12.578850957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:02:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:05.554552456 +0000 UTC m=+429.913274772" watchObservedRunningTime="2026-10-31 01:03:05.580555443 +0000 UTC m=+429.939277761" Oct 31 01:03:05.605816 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:05.605752 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8" podStartSLOduration=2.154797427 podStartE2EDuration="14.60573691s" totalImagesPullingTime="12.450939483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:02:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:05.580742386 +0000 UTC m=+429.939464702" watchObservedRunningTime="2026-10-31 01:03:05.60573691 +0000 UTC m=+429.964459263" Oct 31 01:03:06.456946 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:06.456688 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" Oct 31 01:03:06.456946 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:06.456899 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" Oct 31 01:03:06.463517 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:06.463489 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" Oct 31 01:03:06.550691 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:06.550657 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8" Oct 31 01:03:06.551728 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:06.551702 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" Oct 31 01:03:06.552543 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:06.552369 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8" Oct 31 01:03:07.571896 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:07.571857 4440 generic.go:356] "Generic (PLEG): container finished" podID="885757ee-5daa-4621-a6b8-68cf24a7b4c5" containerID="cfeee33255a3a121043fa0ef4a822ed82f8e301e6749707bcbbf250a89c2f5e9" exitCode=0 Oct 31 01:03:07.572648 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:07.572610 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lcqfh" event={"ID":"885757ee-5daa-4621-a6b8-68cf24a7b4c5","Type":"ContainerDied","Data":"cfeee33255a3a121043fa0ef4a822ed82f8e301e6749707bcbbf250a89c2f5e9"} Oct 31 01:03:08.586490 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:08.586120 4440 generic.go:356] "Generic (PLEG): container finished" podID="3da0ab7f-932f-4367-a609-fe25d23cac66" containerID="2036bb2b572010bdb37b85c694caf72495747cbc4845fdff3aef6af801bc8e83" exitCode=0 Oct 31 01:03:08.588001 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:08.587967 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wb9gd" event={"ID":"3da0ab7f-932f-4367-a609-fe25d23cac66","Type":"ContainerDied","Data":"2036bb2b572010bdb37b85c694caf72495747cbc4845fdff3aef6af801bc8e83"} Oct 31 01:03:10.607308 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:10.607268 4440 generic.go:356] "Generic (PLEG): container finished" podID="e05eb907-6ddc-4767-9d8b-aad5f0ed41e1" containerID="77d8e62cfa9d26d78cc46481196be38e39c1e13b4d61ed6de1feb2f5cb8d25d3" exitCode=0 Oct 31 01:03:10.607787 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:10.607381 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxdnw9" event={"ID":"e05eb907-6ddc-4767-9d8b-aad5f0ed41e1","Type":"ContainerDied","Data":"77d8e62cfa9d26d78cc46481196be38e39c1e13b4d61ed6de1feb2f5cb8d25d3"} Oct 31 01:03:11.615988 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:11.615227 4440 generic.go:356] "Generic (PLEG): container finished" podID="c2ea521c-7769-4b06-9511-c2a63c2e15aa" containerID="3cd20dd2f59172a925a61fb5724f5099e806a3b343843c8f8ac93f48ba74d29c" exitCode=0 Oct 31 01:03:11.615988 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:11.615562 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rfsw9" event={"ID":"c2ea521c-7769-4b06-9511-c2a63c2e15aa","Type":"ContainerDied","Data":"3cd20dd2f59172a925a61fb5724f5099e806a3b343843c8f8ac93f48ba74d29c"} Oct 31 01:03:11.618364 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:11.618248 4440 generic.go:356] "Generic (PLEG): container finished" podID="489ce419-a3c4-4de9-b2a4-4fdaff265be9" containerID="7231d6dbda07c4e71099a02cace02f78f492318d5181a0699155a00c84379e4d" exitCode=0 Oct 31 01:03:11.618364 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:11.618331 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acwd27" event={"ID":"489ce419-a3c4-4de9-b2a4-4fdaff265be9","Type":"ContainerDied","Data":"7231d6dbda07c4e71099a02cace02f78f492318d5181a0699155a00c84379e4d"} Oct 31 01:03:11.621703 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:11.621673 4440 generic.go:356] "Generic (PLEG): container finished" podID="d5148e11-ffda-4e2f-be35-f17c13e0515b" containerID="9d6744895d84dd2738016011748f866097d5abece625d09287df12d4f500a53d" exitCode=0 Oct 31 01:03:11.621795 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:11.621764 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axh5h6" event={"ID":"d5148e11-ffda-4e2f-be35-f17c13e0515b","Type":"ContainerDied","Data":"9d6744895d84dd2738016011748f866097d5abece625d09287df12d4f500a53d"} Oct 31 01:03:11.624176 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:11.624071 4440 generic.go:356] "Generic (PLEG): container finished" podID="7c80a8af-321f-452c-8ee6-cabf283eaf31" containerID="018286127c0d6b8056809837793c5b8c56f67e723a2446890be5a6f621279eaf" exitCode=0 Oct 31 01:03:11.624176 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:11.624150 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9zhn" event={"ID":"7c80a8af-321f-452c-8ee6-cabf283eaf31","Type":"ContainerDied","Data":"018286127c0d6b8056809837793c5b8c56f67e723a2446890be5a6f621279eaf"} Oct 31 01:03:11.627089 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:11.627066 4440 generic.go:356] "Generic (PLEG): container finished" podID="fffeb29e-3429-4a1e-a589-11367e62c490" containerID="807ed0ecf0619948daeddf51ea70d4f942f49240faeb5c147ef11992ddf6b2fe" exitCode=0 Oct 31 01:03:11.627188 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:11.627116 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qjfwr" event={"ID":"fffeb29e-3429-4a1e-a589-11367e62c490","Type":"ContainerDied","Data":"807ed0ecf0619948daeddf51ea70d4f942f49240faeb5c147ef11992ddf6b2fe"} Oct 31 01:03:12.633468 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:12.633409 4440 generic.go:356] "Generic (PLEG): container finished" podID="38588f65-5d75-4b58-a7f6-912cc02c2064" containerID="78fcd923e58da9eac310e00d082f2c4e36e2bb5fde5883ad1f983692338e5608" exitCode=0 Oct 31 01:03:12.633838 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:12.633450 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xltjz" event={"ID":"38588f65-5d75-4b58-a7f6-912cc02c2064","Type":"ContainerDied","Data":"78fcd923e58da9eac310e00d082f2c4e36e2bb5fde5883ad1f983692338e5608"} Oct 31 01:03:12.635277 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:12.635251 4440 generic.go:356] "Generic (PLEG): container finished" podID="bb3f2b0e-88c7-4792-85ff-02c54f46cafd" containerID="043e8ab812736c41629cf8fbd6d8d7f6c700d433812f2e05b6d8e65248a9b7b5" exitCode=0 Oct 31 01:03:12.635385 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:12.635331 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckzpsj" event={"ID":"bb3f2b0e-88c7-4792-85ff-02c54f46cafd","Type":"ContainerDied","Data":"043e8ab812736c41629cf8fbd6d8d7f6c700d433812f2e05b6d8e65248a9b7b5"} Oct 31 01:03:12.636885 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:12.636861 4440 generic.go:356] "Generic (PLEG): container finished" podID="9eb24e27-6ef1-44f1-bde2-130f56377c8c" containerID="907a6c82cdbc4e92d55b2979701884fe637da296a5e0632702ec7483cc54a25a" exitCode=0 Oct 31 01:03:12.636998 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:12.636976 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzqfq" event={"ID":"9eb24e27-6ef1-44f1-bde2-130f56377c8c","Type":"ContainerDied","Data":"907a6c82cdbc4e92d55b2979701884fe637da296a5e0632702ec7483cc54a25a"} Oct 31 01:03:12.638891 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:12.638865 4440 generic.go:356] "Generic (PLEG): container finished" podID="885757ee-5daa-4621-a6b8-68cf24a7b4c5" containerID="91719a60c5f202ef255e5369e9c9dec78a0e2b6616ff49e1dfa792bf4677355f" exitCode=0 Oct 31 01:03:12.639002 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:12.638938 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lcqfh" event={"ID":"885757ee-5daa-4621-a6b8-68cf24a7b4c5","Type":"ContainerDied","Data":"91719a60c5f202ef255e5369e9c9dec78a0e2b6616ff49e1dfa792bf4677355f"} Oct 31 01:03:12.640616 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:12.640595 4440 generic.go:356] "Generic (PLEG): container finished" podID="6193fb88-ac70-40dc-bb5d-e37b6e821959" containerID="ecf3ffd0607d63dee67f3b0c842d3742e9226a1ac354c1c6de7a2a2902319c8b" exitCode=0 Oct 31 01:03:12.640718 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:12.640686 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716w4hx" event={"ID":"6193fb88-ac70-40dc-bb5d-e37b6e821959","Type":"ContainerDied","Data":"ecf3ffd0607d63dee67f3b0c842d3742e9226a1ac354c1c6de7a2a2902319c8b"} Oct 31 01:03:13.651216 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:13.651174 4440 generic.go:356] "Generic (PLEG): container finished" podID="489ce419-a3c4-4de9-b2a4-4fdaff265be9" containerID="4a0264de2d84a174215e36ab01b701ca25732db272e4257614c1fc42a3ddb4c9" exitCode=0 Oct 31 01:03:13.651572 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:13.651269 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acwd27" event={"ID":"489ce419-a3c4-4de9-b2a4-4fdaff265be9","Type":"ContainerDied","Data":"4a0264de2d84a174215e36ab01b701ca25732db272e4257614c1fc42a3ddb4c9"} Oct 31 01:03:13.659286 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:13.659210 4440 generic.go:356] "Generic (PLEG): container finished" podID="d5148e11-ffda-4e2f-be35-f17c13e0515b" containerID="2833fefedc79918b06fe328f64272a5c982e95b49aba77bc15458fe97646a3d8" exitCode=0 Oct 31 01:03:13.659394 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:13.659275 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axh5h6" event={"ID":"d5148e11-ffda-4e2f-be35-f17c13e0515b","Type":"ContainerDied","Data":"2833fefedc79918b06fe328f64272a5c982e95b49aba77bc15458fe97646a3d8"} Oct 31 01:03:13.661274 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:13.661253 4440 generic.go:356] "Generic (PLEG): container finished" podID="7c80a8af-321f-452c-8ee6-cabf283eaf31" containerID="badb6c2555e68523ca4736449f759b22eeff4c7e3b50668f8db9efdeea49b0fc" exitCode=0 Oct 31 01:03:13.661360 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:13.661333 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9zhn" event={"ID":"7c80a8af-321f-452c-8ee6-cabf283eaf31","Type":"ContainerDied","Data":"badb6c2555e68523ca4736449f759b22eeff4c7e3b50668f8db9efdeea49b0fc"} Oct 31 01:03:13.663379 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:13.663359 4440 generic.go:356] "Generic (PLEG): container finished" podID="fffeb29e-3429-4a1e-a589-11367e62c490" containerID="6ff5cf31551c4448a7fed89607e9518bcc5b6372d3c8f850c6b34fb2fdb27d95" exitCode=0 Oct 31 01:03:13.663527 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:13.663456 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qjfwr" event={"ID":"fffeb29e-3429-4a1e-a589-11367e62c490","Type":"ContainerDied","Data":"6ff5cf31551c4448a7fed89607e9518bcc5b6372d3c8f850c6b34fb2fdb27d95"} Oct 31 01:03:13.665682 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:13.665659 4440 generic.go:356] "Generic (PLEG): container finished" podID="38588f65-5d75-4b58-a7f6-912cc02c2064" containerID="a6434201fcd0bec6440c0e483ae6ea3eb129c65f1690ba9eeba8dc82dcda2dfb" exitCode=0 Oct 31 01:03:13.665788 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:13.665745 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xltjz" event={"ID":"38588f65-5d75-4b58-a7f6-912cc02c2064","Type":"ContainerDied","Data":"a6434201fcd0bec6440c0e483ae6ea3eb129c65f1690ba9eeba8dc82dcda2dfb"} Oct 31 01:03:13.669865 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:13.669841 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzqfq" event={"ID":"9eb24e27-6ef1-44f1-bde2-130f56377c8c","Type":"ContainerStarted","Data":"f4dd27563d9db6160a741256e77871851f84fbadb8ca62e57fe1a687a8facfed"} Oct 31 01:03:13.672088 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:13.672069 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxdnw9" event={"ID":"e05eb907-6ddc-4767-9d8b-aad5f0ed41e1","Type":"ContainerStarted","Data":"06c2c9e71a0b1e3f7051404c708de3462c631d7aa286ae85ce4b9a72edcc9b1f"} Oct 31 01:03:13.673925 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:13.673907 4440 generic.go:356] "Generic (PLEG): container finished" podID="6193fb88-ac70-40dc-bb5d-e37b6e821959" containerID="d41d768c091e80d2f7c68881f22ee619a9730ea46ceb9d837b137633c2cccdfe" exitCode=0 Oct 31 01:03:13.673996 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:13.673979 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716w4hx" event={"ID":"6193fb88-ac70-40dc-bb5d-e37b6e821959","Type":"ContainerDied","Data":"d41d768c091e80d2f7c68881f22ee619a9730ea46ceb9d837b137633c2cccdfe"} Oct 31 01:03:13.676182 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:13.676159 4440 generic.go:356] "Generic (PLEG): container finished" podID="3da0ab7f-932f-4367-a609-fe25d23cac66" containerID="29f837bc723f6140041d76e691f40cfec99fa6f822395e7ff443815c57182896" exitCode=0 Oct 31 01:03:13.676241 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:13.676177 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wb9gd" event={"ID":"3da0ab7f-932f-4367-a609-fe25d23cac66","Type":"ContainerDied","Data":"29f837bc723f6140041d76e691f40cfec99fa6f822395e7ff443815c57182896"} Oct 31 01:03:14.691089 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:14.691060 4440 generic.go:356] "Generic (PLEG): container finished" podID="e05eb907-6ddc-4767-9d8b-aad5f0ed41e1" containerID="06c2c9e71a0b1e3f7051404c708de3462c631d7aa286ae85ce4b9a72edcc9b1f" exitCode=0 Oct 31 01:03:14.691512 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:14.691256 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxdnw9" event={"ID":"e05eb907-6ddc-4767-9d8b-aad5f0ed41e1","Type":"ContainerDied","Data":"06c2c9e71a0b1e3f7051404c708de3462c631d7aa286ae85ce4b9a72edcc9b1f"} Oct 31 01:03:14.700260 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:14.699316 4440 generic.go:356] "Generic (PLEG): container finished" podID="9eb24e27-6ef1-44f1-bde2-130f56377c8c" containerID="f4dd27563d9db6160a741256e77871851f84fbadb8ca62e57fe1a687a8facfed" exitCode=0 Oct 31 01:03:14.700260 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:14.699360 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzqfq" event={"ID":"9eb24e27-6ef1-44f1-bde2-130f56377c8c","Type":"ContainerDied","Data":"f4dd27563d9db6160a741256e77871851f84fbadb8ca62e57fe1a687a8facfed"} Oct 31 01:03:15.722435 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:15.722363 4440 generic.go:356] "Generic (PLEG): container finished" podID="c2ea521c-7769-4b06-9511-c2a63c2e15aa" containerID="ad478e426f4af911f8c23ab8c63010d98ca41d6e7211c819b39d71bf0b2ebbab" exitCode=0 Oct 31 01:03:15.723087 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:15.722532 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rfsw9" event={"ID":"c2ea521c-7769-4b06-9511-c2a63c2e15aa","Type":"ContainerDied","Data":"ad478e426f4af911f8c23ab8c63010d98ca41d6e7211c819b39d71bf0b2ebbab"} Oct 31 01:03:26.937778 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:26.934892 4440 generic.go:356] "Generic (PLEG): container finished" podID="bb3f2b0e-88c7-4792-85ff-02c54f46cafd" containerID="d8a684734c64a23ca27bca54e98d9814515cf67e83908532576e84c88c3d01e1" exitCode=0 Oct 31 01:03:26.937778 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:26.934966 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckzpsj" event={"ID":"bb3f2b0e-88c7-4792-85ff-02c54f46cafd","Type":"ContainerDied","Data":"d8a684734c64a23ca27bca54e98d9814515cf67e83908532576e84c88c3d01e1"} Oct 31 01:03:31.050206 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:03:31.050163 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:03:35.062231 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:35.062112 4440 generic.go:356] "Generic (PLEG): container finished" podID="885757ee-5daa-4621-a6b8-68cf24a7b4c5" containerID="04d587ce166f4eb9d7ceccdd388e8526b68153c72a9d7501f402043f7b862edb" exitCode=0 Oct 31 01:03:35.062764 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:35.062240 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lcqfh" event={"ID":"885757ee-5daa-4621-a6b8-68cf24a7b4c5","Type":"ContainerDied","Data":"04d587ce166f4eb9d7ceccdd388e8526b68153c72a9d7501f402043f7b862edb"} Oct 31 01:03:36.338292 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:36.338221 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lcqfh" Oct 31 01:03:36.472243 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:36.472186 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/885757ee-5daa-4621-a6b8-68cf24a7b4c5-util\" (UniqueName: \"kubernetes.io/empty-dir/885757ee-5daa-4621-a6b8-68cf24a7b4c5-util\") pod \"885757ee-5daa-4621-a6b8-68cf24a7b4c5\" (UID: \"885757ee-5daa-4621-a6b8-68cf24a7b4c5\") " Oct 31 01:03:36.472555 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:36.472255 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/885757ee-5daa-4621-a6b8-68cf24a7b4c5-kube-api-access-2x8xx\" (UniqueName: \"kubernetes.io/projected/885757ee-5daa-4621-a6b8-68cf24a7b4c5-kube-api-access-2x8xx\") pod \"885757ee-5daa-4621-a6b8-68cf24a7b4c5\" (UID: \"885757ee-5daa-4621-a6b8-68cf24a7b4c5\") " Oct 31 01:03:36.472555 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:36.472313 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/885757ee-5daa-4621-a6b8-68cf24a7b4c5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/885757ee-5daa-4621-a6b8-68cf24a7b4c5-bundle\") pod \"885757ee-5daa-4621-a6b8-68cf24a7b4c5\" (UID: \"885757ee-5daa-4621-a6b8-68cf24a7b4c5\") " Oct 31 01:03:36.473357 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:36.472750 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/885757ee-5daa-4621-a6b8-68cf24a7b4c5-bundle" pod "885757ee-5daa-4621-a6b8-68cf24a7b4c5" (UID: "885757ee-5daa-4621-a6b8-68cf24a7b4c5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:36.479172 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:36.478948 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/885757ee-5daa-4621-a6b8-68cf24a7b4c5-util" pod "885757ee-5daa-4621-a6b8-68cf24a7b4c5" (UID: "885757ee-5daa-4621-a6b8-68cf24a7b4c5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:36.513410 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:36.513369 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/885757ee-5daa-4621-a6b8-68cf24a7b4c5-kube-api-access-2x8xx" pod "885757ee-5daa-4621-a6b8-68cf24a7b4c5" (UID: "885757ee-5daa-4621-a6b8-68cf24a7b4c5"). InnerVolumeSpecName "kube-api-access-2x8xx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:03:36.573627 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:36.573595 4440 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/885757ee-5daa-4621-a6b8-68cf24a7b4c5-util\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:36.573627 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:36.573630 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x8xx\" (UniqueName: \"kubernetes.io/projected/885757ee-5daa-4621-a6b8-68cf24a7b4c5-kube-api-access-2x8xx\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:36.573940 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:36.573643 4440 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/885757ee-5daa-4621-a6b8-68cf24a7b4c5-bundle\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:37.099295 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:37.099257 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lcqfh" event={"ID":"885757ee-5daa-4621-a6b8-68cf24a7b4c5","Type":"ContainerDied","Data":"60e7f82deaa5f955bcf970be7a32edf04cb6c20613e55be33021b904c96b9cbf"} Oct 31 01:03:37.099608 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:37.099318 4440 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60e7f82deaa5f955bcf970be7a32edf04cb6c20613e55be33021b904c96b9cbf" Oct 31 01:03:37.099608 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:37.099398 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lcqfh" Oct 31 01:03:38.123727 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:38.123319 4440 generic.go:356] "Generic (PLEG): container finished" podID="38588f65-5d75-4b58-a7f6-912cc02c2064" containerID="8833df458f608e4135024d2c75179cacae869a0b29e9feb819634aa337f46989" exitCode=0 Oct 31 01:03:38.123727 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:38.123439 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xltjz" event={"ID":"38588f65-5d75-4b58-a7f6-912cc02c2064","Type":"ContainerDied","Data":"8833df458f608e4135024d2c75179cacae869a0b29e9feb819634aa337f46989"} Oct 31 01:03:39.131576 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:39.131540 4440 generic.go:356] "Generic (PLEG): container finished" podID="6193fb88-ac70-40dc-bb5d-e37b6e821959" containerID="ea645bcf776d67bf29083b81bf7462ce688b283d583d1b2caa4185f6979dd9bf" exitCode=0 Oct 31 01:03:39.132027 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:39.131615 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716w4hx" event={"ID":"6193fb88-ac70-40dc-bb5d-e37b6e821959","Type":"ContainerDied","Data":"ea645bcf776d67bf29083b81bf7462ce688b283d583d1b2caa4185f6979dd9bf"} Oct 31 01:03:39.134716 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:39.134686 4440 generic.go:356] "Generic (PLEG): container finished" podID="d5148e11-ffda-4e2f-be35-f17c13e0515b" containerID="d9050b3187201940e4bda84b64d29aa2638f994ab2a788c1f49d9ad2935ef72a" exitCode=0 Oct 31 01:03:39.134873 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:39.134772 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axh5h6" event={"ID":"d5148e11-ffda-4e2f-be35-f17c13e0515b","Type":"ContainerDied","Data":"d9050b3187201940e4bda84b64d29aa2638f994ab2a788c1f49d9ad2935ef72a"} Oct 31 01:03:39.138196 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:39.138167 4440 generic.go:356] "Generic (PLEG): container finished" podID="7c80a8af-321f-452c-8ee6-cabf283eaf31" containerID="0d2f78c46ee30af14b5e745641e2b21cfe84f4b26062140b71a09e08b77cc4fe" exitCode=0 Oct 31 01:03:39.138323 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:39.138267 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9zhn" event={"ID":"7c80a8af-321f-452c-8ee6-cabf283eaf31","Type":"ContainerDied","Data":"0d2f78c46ee30af14b5e745641e2b21cfe84f4b26062140b71a09e08b77cc4fe"} Oct 31 01:03:39.141573 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:39.141545 4440 generic.go:356] "Generic (PLEG): container finished" podID="fffeb29e-3429-4a1e-a589-11367e62c490" containerID="feac5b50b86040445602d3c8fff3e8c9e7f910ff8c842c835176603b767b2eac" exitCode=0 Oct 31 01:03:39.141691 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:39.141656 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qjfwr" event={"ID":"fffeb29e-3429-4a1e-a589-11367e62c490","Type":"ContainerDied","Data":"feac5b50b86040445602d3c8fff3e8c9e7f910ff8c842c835176603b767b2eac"} Oct 31 01:03:39.144331 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:39.144253 4440 generic.go:356] "Generic (PLEG): container finished" podID="e05eb907-6ddc-4767-9d8b-aad5f0ed41e1" containerID="ab4f9515e225b36f7f65172c0896fa7692efbe80fd2c727c815e5e91f8b8436d" exitCode=0 Oct 31 01:03:39.144331 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:39.144283 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxdnw9" event={"ID":"e05eb907-6ddc-4767-9d8b-aad5f0ed41e1","Type":"ContainerDied","Data":"ab4f9515e225b36f7f65172c0896fa7692efbe80fd2c727c815e5e91f8b8436d"} Oct 31 01:03:39.358378 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:39.358352 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xltjz" Oct 31 01:03:39.500933 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:39.500898 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38588f65-5d75-4b58-a7f6-912cc02c2064-bundle\" (UniqueName: \"kubernetes.io/empty-dir/38588f65-5d75-4b58-a7f6-912cc02c2064-bundle\") pod \"38588f65-5d75-4b58-a7f6-912cc02c2064\" (UID: \"38588f65-5d75-4b58-a7f6-912cc02c2064\") " Oct 31 01:03:39.501110 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:39.500947 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38588f65-5d75-4b58-a7f6-912cc02c2064-kube-api-access-rtbfv\" (UniqueName: \"kubernetes.io/projected/38588f65-5d75-4b58-a7f6-912cc02c2064-kube-api-access-rtbfv\") pod \"38588f65-5d75-4b58-a7f6-912cc02c2064\" (UID: \"38588f65-5d75-4b58-a7f6-912cc02c2064\") " Oct 31 01:03:39.501110 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:39.501020 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38588f65-5d75-4b58-a7f6-912cc02c2064-util\" (UniqueName: \"kubernetes.io/empty-dir/38588f65-5d75-4b58-a7f6-912cc02c2064-util\") pod \"38588f65-5d75-4b58-a7f6-912cc02c2064\" (UID: \"38588f65-5d75-4b58-a7f6-912cc02c2064\") " Oct 31 01:03:39.501402 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:39.501378 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38588f65-5d75-4b58-a7f6-912cc02c2064-bundle" pod "38588f65-5d75-4b58-a7f6-912cc02c2064" (UID: "38588f65-5d75-4b58-a7f6-912cc02c2064"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:39.503086 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:39.503060 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38588f65-5d75-4b58-a7f6-912cc02c2064-kube-api-access-rtbfv" pod "38588f65-5d75-4b58-a7f6-912cc02c2064" (UID: "38588f65-5d75-4b58-a7f6-912cc02c2064"). InnerVolumeSpecName "kube-api-access-rtbfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:03:39.506453 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:39.506429 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38588f65-5d75-4b58-a7f6-912cc02c2064-util" pod "38588f65-5d75-4b58-a7f6-912cc02c2064" (UID: "38588f65-5d75-4b58-a7f6-912cc02c2064"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:39.601750 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:39.601680 4440 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38588f65-5d75-4b58-a7f6-912cc02c2064-util\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:39.601750 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:39.601706 4440 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38588f65-5d75-4b58-a7f6-912cc02c2064-bundle\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:39.601750 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:39.601717 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtbfv\" (UniqueName: \"kubernetes.io/projected/38588f65-5d75-4b58-a7f6-912cc02c2064-kube-api-access-rtbfv\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:40.150278 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.150182 4440 generic.go:356] "Generic (PLEG): container finished" podID="bb3f2b0e-88c7-4792-85ff-02c54f46cafd" containerID="8b823a3b7b9fd4a7db88c893d66ddacf207962cedf5354d4b07f73a95909c659" exitCode=0 Oct 31 01:03:40.150278 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.150260 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckzpsj" event={"ID":"bb3f2b0e-88c7-4792-85ff-02c54f46cafd","Type":"ContainerDied","Data":"8b823a3b7b9fd4a7db88c893d66ddacf207962cedf5354d4b07f73a95909c659"} Oct 31 01:03:40.152453 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.152430 4440 generic.go:356] "Generic (PLEG): container finished" podID="9eb24e27-6ef1-44f1-bde2-130f56377c8c" containerID="9c4349f7aafd04e43276a978a1b95a765b0a6a93f37d8c4567b6e1e4cdad6631" exitCode=0 Oct 31 01:03:40.152586 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.152494 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzqfq" event={"ID":"9eb24e27-6ef1-44f1-bde2-130f56377c8c","Type":"ContainerDied","Data":"9c4349f7aafd04e43276a978a1b95a765b0a6a93f37d8c4567b6e1e4cdad6631"} Oct 31 01:03:40.154581 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.154561 4440 generic.go:356] "Generic (PLEG): container finished" podID="3da0ab7f-932f-4367-a609-fe25d23cac66" containerID="5a8227ec9021414e70c4cc353a3db91d51bf06e719552f5b248dc8b3cd681c52" exitCode=0 Oct 31 01:03:40.154679 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.154648 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wb9gd" event={"ID":"3da0ab7f-932f-4367-a609-fe25d23cac66","Type":"ContainerDied","Data":"5a8227ec9021414e70c4cc353a3db91d51bf06e719552f5b248dc8b3cd681c52"} Oct 31 01:03:40.156585 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.156569 4440 generic.go:356] "Generic (PLEG): container finished" podID="c2ea521c-7769-4b06-9511-c2a63c2e15aa" containerID="7226f39264ab023eb81ef94ea5d8f0c62b3ce26b6cc78610b4732d742143fab2" exitCode=0 Oct 31 01:03:40.156634 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.156626 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rfsw9" event={"ID":"c2ea521c-7769-4b06-9511-c2a63c2e15aa","Type":"ContainerDied","Data":"7226f39264ab023eb81ef94ea5d8f0c62b3ce26b6cc78610b4732d742143fab2"} Oct 31 01:03:40.158513 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.158496 4440 generic.go:356] "Generic (PLEG): container finished" podID="489ce419-a3c4-4de9-b2a4-4fdaff265be9" containerID="f91a26592bc24502007796a3a883944b0b2c80e40361eb2d10666586ee3f90c3" exitCode=0 Oct 31 01:03:40.158599 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.158580 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acwd27" event={"ID":"489ce419-a3c4-4de9-b2a4-4fdaff265be9","Type":"ContainerDied","Data":"f91a26592bc24502007796a3a883944b0b2c80e40361eb2d10666586ee3f90c3"} Oct 31 01:03:40.160334 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.160267 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xltjz" Oct 31 01:03:40.160334 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.160275 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xltjz" event={"ID":"38588f65-5d75-4b58-a7f6-912cc02c2064","Type":"ContainerDied","Data":"bd0eba94547e3d1cf59f2432b41ffc1b68ca74142ef6a5e5596a636e4e3fcbbd"} Oct 31 01:03:40.160334 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.160316 4440 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd0eba94547e3d1cf59f2432b41ffc1b68ca74142ef6a5e5596a636e4e3fcbbd" Oct 31 01:03:40.335765 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.335740 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxdnw9" Oct 31 01:03:40.353718 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.353696 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qjfwr" Oct 31 01:03:40.363971 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.363938 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716w4hx" Oct 31 01:03:40.367728 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.367710 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axh5h6" Oct 31 01:03:40.373031 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.373010 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9zhn" Oct 31 01:03:40.511125 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.511094 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e05eb907-6ddc-4767-9d8b-aad5f0ed41e1-kube-api-access-vxr76\" (UniqueName: \"kubernetes.io/projected/e05eb907-6ddc-4767-9d8b-aad5f0ed41e1-kube-api-access-vxr76\") pod \"e05eb907-6ddc-4767-9d8b-aad5f0ed41e1\" (UID: \"e05eb907-6ddc-4767-9d8b-aad5f0ed41e1\") " Oct 31 01:03:40.511125 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.511130 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6193fb88-ac70-40dc-bb5d-e37b6e821959-kube-api-access-2799t\" (UniqueName: \"kubernetes.io/projected/6193fb88-ac70-40dc-bb5d-e37b6e821959-kube-api-access-2799t\") pod \"6193fb88-ac70-40dc-bb5d-e37b6e821959\" (UID: \"6193fb88-ac70-40dc-bb5d-e37b6e821959\") " Oct 31 01:03:40.511395 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.511159 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fffeb29e-3429-4a1e-a589-11367e62c490-util\" (UniqueName: \"kubernetes.io/empty-dir/fffeb29e-3429-4a1e-a589-11367e62c490-util\") pod \"fffeb29e-3429-4a1e-a589-11367e62c490\" (UID: \"fffeb29e-3429-4a1e-a589-11367e62c490\") " Oct 31 01:03:40.511395 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.511193 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e05eb907-6ddc-4767-9d8b-aad5f0ed41e1-util\" (UniqueName: \"kubernetes.io/empty-dir/e05eb907-6ddc-4767-9d8b-aad5f0ed41e1-util\") pod \"e05eb907-6ddc-4767-9d8b-aad5f0ed41e1\" (UID: \"e05eb907-6ddc-4767-9d8b-aad5f0ed41e1\") " Oct 31 01:03:40.511395 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.511220 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c80a8af-321f-452c-8ee6-cabf283eaf31-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c80a8af-321f-452c-8ee6-cabf283eaf31-bundle\") pod \"7c80a8af-321f-452c-8ee6-cabf283eaf31\" (UID: \"7c80a8af-321f-452c-8ee6-cabf283eaf31\") " Oct 31 01:03:40.511395 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.511243 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e05eb907-6ddc-4767-9d8b-aad5f0ed41e1-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e05eb907-6ddc-4767-9d8b-aad5f0ed41e1-bundle\") pod \"e05eb907-6ddc-4767-9d8b-aad5f0ed41e1\" (UID: \"e05eb907-6ddc-4767-9d8b-aad5f0ed41e1\") " Oct 31 01:03:40.511395 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.511260 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fffeb29e-3429-4a1e-a589-11367e62c490-kube-api-access-ftl6b\" (UniqueName: \"kubernetes.io/projected/fffeb29e-3429-4a1e-a589-11367e62c490-kube-api-access-ftl6b\") pod \"fffeb29e-3429-4a1e-a589-11367e62c490\" (UID: \"fffeb29e-3429-4a1e-a589-11367e62c490\") " Oct 31 01:03:40.511395 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.511284 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5148e11-ffda-4e2f-be35-f17c13e0515b-util\" (UniqueName: \"kubernetes.io/empty-dir/d5148e11-ffda-4e2f-be35-f17c13e0515b-util\") pod \"d5148e11-ffda-4e2f-be35-f17c13e0515b\" (UID: \"d5148e11-ffda-4e2f-be35-f17c13e0515b\") " Oct 31 01:03:40.511395 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.511319 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c80a8af-321f-452c-8ee6-cabf283eaf31-util\" (UniqueName: \"kubernetes.io/empty-dir/7c80a8af-321f-452c-8ee6-cabf283eaf31-util\") pod \"7c80a8af-321f-452c-8ee6-cabf283eaf31\" (UID: \"7c80a8af-321f-452c-8ee6-cabf283eaf31\") " Oct 31 01:03:40.511395 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.511345 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6193fb88-ac70-40dc-bb5d-e37b6e821959-util\" (UniqueName: \"kubernetes.io/empty-dir/6193fb88-ac70-40dc-bb5d-e37b6e821959-util\") pod \"6193fb88-ac70-40dc-bb5d-e37b6e821959\" (UID: \"6193fb88-ac70-40dc-bb5d-e37b6e821959\") " Oct 31 01:03:40.511395 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.511382 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c80a8af-321f-452c-8ee6-cabf283eaf31-kube-api-access-jdjfm\" (UniqueName: \"kubernetes.io/projected/7c80a8af-321f-452c-8ee6-cabf283eaf31-kube-api-access-jdjfm\") pod \"7c80a8af-321f-452c-8ee6-cabf283eaf31\" (UID: \"7c80a8af-321f-452c-8ee6-cabf283eaf31\") " Oct 31 01:03:40.511841 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.511472 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6193fb88-ac70-40dc-bb5d-e37b6e821959-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6193fb88-ac70-40dc-bb5d-e37b6e821959-bundle\") pod \"6193fb88-ac70-40dc-bb5d-e37b6e821959\" (UID: \"6193fb88-ac70-40dc-bb5d-e37b6e821959\") " Oct 31 01:03:40.511841 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.511510 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5148e11-ffda-4e2f-be35-f17c13e0515b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5148e11-ffda-4e2f-be35-f17c13e0515b-bundle\") pod \"d5148e11-ffda-4e2f-be35-f17c13e0515b\" (UID: \"d5148e11-ffda-4e2f-be35-f17c13e0515b\") " Oct 31 01:03:40.511841 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.511538 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5148e11-ffda-4e2f-be35-f17c13e0515b-kube-api-access-qq7xr\" (UniqueName: \"kubernetes.io/projected/d5148e11-ffda-4e2f-be35-f17c13e0515b-kube-api-access-qq7xr\") pod \"d5148e11-ffda-4e2f-be35-f17c13e0515b\" (UID: \"d5148e11-ffda-4e2f-be35-f17c13e0515b\") " Oct 31 01:03:40.511841 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.511570 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fffeb29e-3429-4a1e-a589-11367e62c490-bundle\" (UniqueName: \"kubernetes.io/empty-dir/fffeb29e-3429-4a1e-a589-11367e62c490-bundle\") pod \"fffeb29e-3429-4a1e-a589-11367e62c490\" (UID: \"fffeb29e-3429-4a1e-a589-11367e62c490\") " Oct 31 01:03:40.511841 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.511754 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c80a8af-321f-452c-8ee6-cabf283eaf31-bundle" pod "7c80a8af-321f-452c-8ee6-cabf283eaf31" (UID: "7c80a8af-321f-452c-8ee6-cabf283eaf31"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:40.512070 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.511846 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e05eb907-6ddc-4767-9d8b-aad5f0ed41e1-bundle" pod "e05eb907-6ddc-4767-9d8b-aad5f0ed41e1" (UID: "e05eb907-6ddc-4767-9d8b-aad5f0ed41e1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:40.512070 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.511866 4440 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c80a8af-321f-452c-8ee6-cabf283eaf31-bundle\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:40.512314 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.512289 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fffeb29e-3429-4a1e-a589-11367e62c490-bundle" pod "fffeb29e-3429-4a1e-a589-11367e62c490" (UID: "fffeb29e-3429-4a1e-a589-11367e62c490"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:40.512919 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.512891 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5148e11-ffda-4e2f-be35-f17c13e0515b-bundle" pod "d5148e11-ffda-4e2f-be35-f17c13e0515b" (UID: "d5148e11-ffda-4e2f-be35-f17c13e0515b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:40.513798 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.513770 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6193fb88-ac70-40dc-bb5d-e37b6e821959-kube-api-access-2799t" pod "6193fb88-ac70-40dc-bb5d-e37b6e821959" (UID: "6193fb88-ac70-40dc-bb5d-e37b6e821959"). InnerVolumeSpecName "kube-api-access-2799t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:03:40.513917 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.513800 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6193fb88-ac70-40dc-bb5d-e37b6e821959-bundle" pod "6193fb88-ac70-40dc-bb5d-e37b6e821959" (UID: "6193fb88-ac70-40dc-bb5d-e37b6e821959"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:40.514743 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.514703 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fffeb29e-3429-4a1e-a589-11367e62c490-kube-api-access-ftl6b" pod "fffeb29e-3429-4a1e-a589-11367e62c490" (UID: "fffeb29e-3429-4a1e-a589-11367e62c490"). InnerVolumeSpecName "kube-api-access-ftl6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:03:40.514829 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.514745 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e05eb907-6ddc-4767-9d8b-aad5f0ed41e1-kube-api-access-vxr76" pod "e05eb907-6ddc-4767-9d8b-aad5f0ed41e1" (UID: "e05eb907-6ddc-4767-9d8b-aad5f0ed41e1"). InnerVolumeSpecName "kube-api-access-vxr76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:03:40.521711 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.521689 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fffeb29e-3429-4a1e-a589-11367e62c490-util" pod "fffeb29e-3429-4a1e-a589-11367e62c490" (UID: "fffeb29e-3429-4a1e-a589-11367e62c490"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:40.522341 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.522318 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c80a8af-321f-452c-8ee6-cabf283eaf31-kube-api-access-jdjfm" pod "7c80a8af-321f-452c-8ee6-cabf283eaf31" (UID: "7c80a8af-321f-452c-8ee6-cabf283eaf31"). InnerVolumeSpecName "kube-api-access-jdjfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:03:40.522624 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.522599 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5148e11-ffda-4e2f-be35-f17c13e0515b-kube-api-access-qq7xr" pod "d5148e11-ffda-4e2f-be35-f17c13e0515b" (UID: "d5148e11-ffda-4e2f-be35-f17c13e0515b"). InnerVolumeSpecName "kube-api-access-qq7xr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:03:40.522729 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.522707 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c80a8af-321f-452c-8ee6-cabf283eaf31-util" pod "7c80a8af-321f-452c-8ee6-cabf283eaf31" (UID: "7c80a8af-321f-452c-8ee6-cabf283eaf31"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:40.523779 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.523760 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6193fb88-ac70-40dc-bb5d-e37b6e821959-util" pod "6193fb88-ac70-40dc-bb5d-e37b6e821959" (UID: "6193fb88-ac70-40dc-bb5d-e37b6e821959"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:40.523948 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.523932 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5148e11-ffda-4e2f-be35-f17c13e0515b-util" pod "d5148e11-ffda-4e2f-be35-f17c13e0515b" (UID: "d5148e11-ffda-4e2f-be35-f17c13e0515b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:40.524160 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.524142 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e05eb907-6ddc-4767-9d8b-aad5f0ed41e1-util" pod "e05eb907-6ddc-4767-9d8b-aad5f0ed41e1" (UID: "e05eb907-6ddc-4767-9d8b-aad5f0ed41e1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:40.613139 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.613112 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxr76\" (UniqueName: \"kubernetes.io/projected/e05eb907-6ddc-4767-9d8b-aad5f0ed41e1-kube-api-access-vxr76\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:40.613139 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.613136 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2799t\" (UniqueName: \"kubernetes.io/projected/6193fb88-ac70-40dc-bb5d-e37b6e821959-kube-api-access-2799t\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:40.613268 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.613146 4440 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fffeb29e-3429-4a1e-a589-11367e62c490-util\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:40.613268 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.613155 4440 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e05eb907-6ddc-4767-9d8b-aad5f0ed41e1-util\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:40.613268 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.613164 4440 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e05eb907-6ddc-4767-9d8b-aad5f0ed41e1-bundle\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:40.613268 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.613173 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftl6b\" (UniqueName: \"kubernetes.io/projected/fffeb29e-3429-4a1e-a589-11367e62c490-kube-api-access-ftl6b\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:40.613268 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.613186 4440 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5148e11-ffda-4e2f-be35-f17c13e0515b-util\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:40.613268 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.613197 4440 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c80a8af-321f-452c-8ee6-cabf283eaf31-util\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:40.613268 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.613204 4440 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6193fb88-ac70-40dc-bb5d-e37b6e821959-util\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:40.613268 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.613212 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdjfm\" (UniqueName: \"kubernetes.io/projected/7c80a8af-321f-452c-8ee6-cabf283eaf31-kube-api-access-jdjfm\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:40.613268 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.613220 4440 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6193fb88-ac70-40dc-bb5d-e37b6e821959-bundle\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:40.613268 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.613228 4440 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5148e11-ffda-4e2f-be35-f17c13e0515b-bundle\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:40.613268 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.613236 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq7xr\" (UniqueName: \"kubernetes.io/projected/d5148e11-ffda-4e2f-be35-f17c13e0515b-kube-api-access-qq7xr\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:40.613268 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:40.613244 4440 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fffeb29e-3429-4a1e-a589-11367e62c490-bundle\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:41.167615 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.167583 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qjfwr" Oct 31 01:03:41.167615 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.167593 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qjfwr" event={"ID":"fffeb29e-3429-4a1e-a589-11367e62c490","Type":"ContainerDied","Data":"db9533e268c4680ffd29272c63ab6c77949f19e7f105c032c156ec1a29e57fda"} Oct 31 01:03:41.168039 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.167639 4440 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db9533e268c4680ffd29272c63ab6c77949f19e7f105c032c156ec1a29e57fda" Oct 31 01:03:41.169459 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.169430 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxdnw9" event={"ID":"e05eb907-6ddc-4767-9d8b-aad5f0ed41e1","Type":"ContainerDied","Data":"14b1f1e26820698192eb659f01dba37b1372a1c9408ae5a9311c3268e9449d02"} Oct 31 01:03:41.169459 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.169464 4440 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14b1f1e26820698192eb659f01dba37b1372a1c9408ae5a9311c3268e9449d02" Oct 31 01:03:41.169661 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.169438 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxdnw9" Oct 31 01:03:41.171296 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.171274 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716w4hx" Oct 31 01:03:41.171444 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.171326 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716w4hx" event={"ID":"6193fb88-ac70-40dc-bb5d-e37b6e821959","Type":"ContainerDied","Data":"e7bc46acc1e50fb98be113b4373e9053dc094582348cffe73364b69e8a245ea8"} Oct 31 01:03:41.171444 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.171364 4440 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7bc46acc1e50fb98be113b4373e9053dc094582348cffe73364b69e8a245ea8" Oct 31 01:03:41.173372 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.173344 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axh5h6" event={"ID":"d5148e11-ffda-4e2f-be35-f17c13e0515b","Type":"ContainerDied","Data":"379b27a0018ea59d9bdb7324efeb1bc8adb94f6d9b9e7824e9b596177952db9c"} Oct 31 01:03:41.173372 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.173362 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axh5h6" Oct 31 01:03:41.173554 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.173378 4440 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="379b27a0018ea59d9bdb7324efeb1bc8adb94f6d9b9e7824e9b596177952db9c" Oct 31 01:03:41.175194 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.175168 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9zhn" event={"ID":"7c80a8af-321f-452c-8ee6-cabf283eaf31","Type":"ContainerDied","Data":"41a1234ceb0fa62f9cea433608f71fedb05b4e0c680904f414fe7ce5f22b495d"} Oct 31 01:03:41.175277 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.175212 4440 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41a1234ceb0fa62f9cea433608f71fedb05b4e0c680904f414fe7ce5f22b495d" Oct 31 01:03:41.175277 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.175218 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9zhn" Oct 31 01:03:41.320259 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.320238 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckzpsj" Oct 31 01:03:41.370484 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.370462 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wb9gd" Oct 31 01:03:41.384753 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.384732 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzqfq" Oct 31 01:03:41.388397 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.387731 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acwd27" Oct 31 01:03:41.394993 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.394976 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rfsw9" Oct 31 01:03:41.422863 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.422806 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb3f2b0e-88c7-4792-85ff-02c54f46cafd-bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb3f2b0e-88c7-4792-85ff-02c54f46cafd-bundle\") pod \"bb3f2b0e-88c7-4792-85ff-02c54f46cafd\" (UID: \"bb3f2b0e-88c7-4792-85ff-02c54f46cafd\") " Oct 31 01:03:41.422863 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.422854 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb3f2b0e-88c7-4792-85ff-02c54f46cafd-util\" (UniqueName: \"kubernetes.io/empty-dir/bb3f2b0e-88c7-4792-85ff-02c54f46cafd-util\") pod \"bb3f2b0e-88c7-4792-85ff-02c54f46cafd\" (UID: \"bb3f2b0e-88c7-4792-85ff-02c54f46cafd\") " Oct 31 01:03:41.423039 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.422979 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb3f2b0e-88c7-4792-85ff-02c54f46cafd-kube-api-access-8rpgc\" (UniqueName: \"kubernetes.io/projected/bb3f2b0e-88c7-4792-85ff-02c54f46cafd-kube-api-access-8rpgc\") pod \"bb3f2b0e-88c7-4792-85ff-02c54f46cafd\" (UID: \"bb3f2b0e-88c7-4792-85ff-02c54f46cafd\") " Oct 31 01:03:41.423595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.423574 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3f2b0e-88c7-4792-85ff-02c54f46cafd-bundle" pod "bb3f2b0e-88c7-4792-85ff-02c54f46cafd" (UID: "bb3f2b0e-88c7-4792-85ff-02c54f46cafd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:41.425014 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.424996 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb3f2b0e-88c7-4792-85ff-02c54f46cafd-kube-api-access-8rpgc" pod "bb3f2b0e-88c7-4792-85ff-02c54f46cafd" (UID: "bb3f2b0e-88c7-4792-85ff-02c54f46cafd"). InnerVolumeSpecName "kube-api-access-8rpgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:03:41.429601 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.429575 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3f2b0e-88c7-4792-85ff-02c54f46cafd-util" pod "bb3f2b0e-88c7-4792-85ff-02c54f46cafd" (UID: "bb3f2b0e-88c7-4792-85ff-02c54f46cafd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:41.523702 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.523670 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/489ce419-a3c4-4de9-b2a4-4fdaff265be9-util\" (UniqueName: \"kubernetes.io/empty-dir/489ce419-a3c4-4de9-b2a4-4fdaff265be9-util\") pod \"489ce419-a3c4-4de9-b2a4-4fdaff265be9\" (UID: \"489ce419-a3c4-4de9-b2a4-4fdaff265be9\") " Oct 31 01:03:41.523871 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.523708 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/489ce419-a3c4-4de9-b2a4-4fdaff265be9-bundle\" (UniqueName: \"kubernetes.io/empty-dir/489ce419-a3c4-4de9-b2a4-4fdaff265be9-bundle\") pod \"489ce419-a3c4-4de9-b2a4-4fdaff265be9\" (UID: \"489ce419-a3c4-4de9-b2a4-4fdaff265be9\") " Oct 31 01:03:41.523871 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.523734 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3da0ab7f-932f-4367-a609-fe25d23cac66-util\" (UniqueName: \"kubernetes.io/empty-dir/3da0ab7f-932f-4367-a609-fe25d23cac66-util\") pod \"3da0ab7f-932f-4367-a609-fe25d23cac66\" (UID: \"3da0ab7f-932f-4367-a609-fe25d23cac66\") " Oct 31 01:03:41.523871 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.523758 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eb24e27-6ef1-44f1-bde2-130f56377c8c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9eb24e27-6ef1-44f1-bde2-130f56377c8c-bundle\") pod \"9eb24e27-6ef1-44f1-bde2-130f56377c8c\" (UID: \"9eb24e27-6ef1-44f1-bde2-130f56377c8c\") " Oct 31 01:03:41.523871 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.523801 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2ea521c-7769-4b06-9511-c2a63c2e15aa-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c2ea521c-7769-4b06-9511-c2a63c2e15aa-bundle\") pod \"c2ea521c-7769-4b06-9511-c2a63c2e15aa\" (UID: \"c2ea521c-7769-4b06-9511-c2a63c2e15aa\") " Oct 31 01:03:41.523871 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.523850 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3da0ab7f-932f-4367-a609-fe25d23cac66-kube-api-access-sp2qm\" (UniqueName: \"kubernetes.io/projected/3da0ab7f-932f-4367-a609-fe25d23cac66-kube-api-access-sp2qm\") pod \"3da0ab7f-932f-4367-a609-fe25d23cac66\" (UID: \"3da0ab7f-932f-4367-a609-fe25d23cac66\") " Oct 31 01:03:41.524135 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.523878 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3da0ab7f-932f-4367-a609-fe25d23cac66-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3da0ab7f-932f-4367-a609-fe25d23cac66-bundle\") pod \"3da0ab7f-932f-4367-a609-fe25d23cac66\" (UID: \"3da0ab7f-932f-4367-a609-fe25d23cac66\") " Oct 31 01:03:41.524135 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.523907 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eb24e27-6ef1-44f1-bde2-130f56377c8c-kube-api-access-kg7sp\" (UniqueName: \"kubernetes.io/projected/9eb24e27-6ef1-44f1-bde2-130f56377c8c-kube-api-access-kg7sp\") pod \"9eb24e27-6ef1-44f1-bde2-130f56377c8c\" (UID: \"9eb24e27-6ef1-44f1-bde2-130f56377c8c\") " Oct 31 01:03:41.524135 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.523935 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2ea521c-7769-4b06-9511-c2a63c2e15aa-kube-api-access-v9bgk\" (UniqueName: \"kubernetes.io/projected/c2ea521c-7769-4b06-9511-c2a63c2e15aa-kube-api-access-v9bgk\") pod \"c2ea521c-7769-4b06-9511-c2a63c2e15aa\" (UID: \"c2ea521c-7769-4b06-9511-c2a63c2e15aa\") " Oct 31 01:03:41.524135 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.523970 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/489ce419-a3c4-4de9-b2a4-4fdaff265be9-kube-api-access-k5ffk\" (UniqueName: \"kubernetes.io/projected/489ce419-a3c4-4de9-b2a4-4fdaff265be9-kube-api-access-k5ffk\") pod \"489ce419-a3c4-4de9-b2a4-4fdaff265be9\" (UID: \"489ce419-a3c4-4de9-b2a4-4fdaff265be9\") " Oct 31 01:03:41.524135 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.524000 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2ea521c-7769-4b06-9511-c2a63c2e15aa-util\" (UniqueName: \"kubernetes.io/empty-dir/c2ea521c-7769-4b06-9511-c2a63c2e15aa-util\") pod \"c2ea521c-7769-4b06-9511-c2a63c2e15aa\" (UID: \"c2ea521c-7769-4b06-9511-c2a63c2e15aa\") " Oct 31 01:03:41.524135 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.524042 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eb24e27-6ef1-44f1-bde2-130f56377c8c-util\" (UniqueName: \"kubernetes.io/empty-dir/9eb24e27-6ef1-44f1-bde2-130f56377c8c-util\") pod \"9eb24e27-6ef1-44f1-bde2-130f56377c8c\" (UID: \"9eb24e27-6ef1-44f1-bde2-130f56377c8c\") " Oct 31 01:03:41.524413 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.524279 4440 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb3f2b0e-88c7-4792-85ff-02c54f46cafd-bundle\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:41.524413 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.524295 4440 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb3f2b0e-88c7-4792-85ff-02c54f46cafd-util\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:41.524413 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.524310 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rpgc\" (UniqueName: \"kubernetes.io/projected/bb3f2b0e-88c7-4792-85ff-02c54f46cafd-kube-api-access-8rpgc\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:41.524588 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.524398 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3da0ab7f-932f-4367-a609-fe25d23cac66-bundle" pod "3da0ab7f-932f-4367-a609-fe25d23cac66" (UID: "3da0ab7f-932f-4367-a609-fe25d23cac66"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:41.524848 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.524495 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2ea521c-7769-4b06-9511-c2a63c2e15aa-bundle" pod "c2ea521c-7769-4b06-9511-c2a63c2e15aa" (UID: "c2ea521c-7769-4b06-9511-c2a63c2e15aa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:41.524982 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.524920 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb24e27-6ef1-44f1-bde2-130f56377c8c-bundle" pod "9eb24e27-6ef1-44f1-bde2-130f56377c8c" (UID: "9eb24e27-6ef1-44f1-bde2-130f56377c8c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:41.525037 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.525004 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/489ce419-a3c4-4de9-b2a4-4fdaff265be9-bundle" pod "489ce419-a3c4-4de9-b2a4-4fdaff265be9" (UID: "489ce419-a3c4-4de9-b2a4-4fdaff265be9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:41.527297 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.527260 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3da0ab7f-932f-4367-a609-fe25d23cac66-kube-api-access-sp2qm" pod "3da0ab7f-932f-4367-a609-fe25d23cac66" (UID: "3da0ab7f-932f-4367-a609-fe25d23cac66"). InnerVolumeSpecName "kube-api-access-sp2qm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:03:41.527404 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.527306 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb24e27-6ef1-44f1-bde2-130f56377c8c-kube-api-access-kg7sp" pod "9eb24e27-6ef1-44f1-bde2-130f56377c8c" (UID: "9eb24e27-6ef1-44f1-bde2-130f56377c8c"). InnerVolumeSpecName "kube-api-access-kg7sp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:03:41.527404 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.527365 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2ea521c-7769-4b06-9511-c2a63c2e15aa-kube-api-access-v9bgk" pod "c2ea521c-7769-4b06-9511-c2a63c2e15aa" (UID: "c2ea521c-7769-4b06-9511-c2a63c2e15aa"). InnerVolumeSpecName "kube-api-access-v9bgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:03:41.527404 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.527391 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/489ce419-a3c4-4de9-b2a4-4fdaff265be9-kube-api-access-k5ffk" pod "489ce419-a3c4-4de9-b2a4-4fdaff265be9" (UID: "489ce419-a3c4-4de9-b2a4-4fdaff265be9"). InnerVolumeSpecName "kube-api-access-k5ffk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:03:41.532965 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.532941 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3da0ab7f-932f-4367-a609-fe25d23cac66-util" pod "3da0ab7f-932f-4367-a609-fe25d23cac66" (UID: "3da0ab7f-932f-4367-a609-fe25d23cac66"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:41.533148 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.533123 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/489ce419-a3c4-4de9-b2a4-4fdaff265be9-util" pod "489ce419-a3c4-4de9-b2a4-4fdaff265be9" (UID: "489ce419-a3c4-4de9-b2a4-4fdaff265be9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:41.533931 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.533909 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2ea521c-7769-4b06-9511-c2a63c2e15aa-util" pod "c2ea521c-7769-4b06-9511-c2a63c2e15aa" (UID: "c2ea521c-7769-4b06-9511-c2a63c2e15aa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:41.534313 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.534296 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb24e27-6ef1-44f1-bde2-130f56377c8c-util" pod "9eb24e27-6ef1-44f1-bde2-130f56377c8c" (UID: "9eb24e27-6ef1-44f1-bde2-130f56377c8c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:03:41.624650 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.624620 4440 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c2ea521c-7769-4b06-9511-c2a63c2e15aa-util\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:41.624650 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.624646 4440 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9eb24e27-6ef1-44f1-bde2-130f56377c8c-util\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:41.624650 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.624655 4440 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/489ce419-a3c4-4de9-b2a4-4fdaff265be9-util\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:41.624848 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.624664 4440 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/489ce419-a3c4-4de9-b2a4-4fdaff265be9-bundle\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:41.624848 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.624672 4440 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3da0ab7f-932f-4367-a609-fe25d23cac66-util\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:41.624848 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.624679 4440 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9eb24e27-6ef1-44f1-bde2-130f56377c8c-bundle\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:41.624848 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.624687 4440 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c2ea521c-7769-4b06-9511-c2a63c2e15aa-bundle\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:41.624848 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.624696 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp2qm\" (UniqueName: \"kubernetes.io/projected/3da0ab7f-932f-4367-a609-fe25d23cac66-kube-api-access-sp2qm\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:41.624848 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.624704 4440 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3da0ab7f-932f-4367-a609-fe25d23cac66-bundle\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:41.624848 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.624713 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg7sp\" (UniqueName: \"kubernetes.io/projected/9eb24e27-6ef1-44f1-bde2-130f56377c8c-kube-api-access-kg7sp\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:41.624848 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.624721 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9bgk\" (UniqueName: \"kubernetes.io/projected/c2ea521c-7769-4b06-9511-c2a63c2e15aa-kube-api-access-v9bgk\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:41.624848 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:41.624730 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5ffk\" (UniqueName: \"kubernetes.io/projected/489ce419-a3c4-4de9-b2a4-4fdaff265be9-kube-api-access-k5ffk\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:42.182798 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:42.182764 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rfsw9" event={"ID":"c2ea521c-7769-4b06-9511-c2a63c2e15aa","Type":"ContainerDied","Data":"3a357b6a0d0d34e773ebac35532ff455fcbbaec95c5a2346a7faa800e12f8500"} Oct 31 01:03:42.182798 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:42.182782 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rfsw9" Oct 31 01:03:42.183269 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:42.182812 4440 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a357b6a0d0d34e773ebac35532ff455fcbbaec95c5a2346a7faa800e12f8500" Oct 31 01:03:42.184800 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:42.184782 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acwd27" Oct 31 01:03:42.185034 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:42.184808 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acwd27" event={"ID":"489ce419-a3c4-4de9-b2a4-4fdaff265be9","Type":"ContainerDied","Data":"6a573376608c60b9a7aa81fa6f77e151c8a7dad55626f1071c2372e2de7ef7ea"} Oct 31 01:03:42.185131 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:42.185050 4440 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a573376608c60b9a7aa81fa6f77e151c8a7dad55626f1071c2372e2de7ef7ea" Oct 31 01:03:42.186847 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:42.186826 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckzpsj" event={"ID":"bb3f2b0e-88c7-4792-85ff-02c54f46cafd","Type":"ContainerDied","Data":"10eceb91d2cda51367e590400a5870683120feb58477fb227502ef6a68d34787"} Oct 31 01:03:42.187005 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:42.186848 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ckzpsj" Oct 31 01:03:42.187130 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:42.186972 4440 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10eceb91d2cda51367e590400a5870683120feb58477fb227502ef6a68d34787" Oct 31 01:03:42.189021 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:42.189006 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzqfq" Oct 31 01:03:42.189021 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:42.189013 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdzqfq" event={"ID":"9eb24e27-6ef1-44f1-bde2-130f56377c8c","Type":"ContainerDied","Data":"a17ccf8cab2705b533a8bbb74b9d5a189fd4b10e85b32ea96d8a29515e4a8992"} Oct 31 01:03:42.189156 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:42.189047 4440 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a17ccf8cab2705b533a8bbb74b9d5a189fd4b10e85b32ea96d8a29515e4a8992" Oct 31 01:03:42.190994 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:42.190974 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wb9gd" event={"ID":"3da0ab7f-932f-4367-a609-fe25d23cac66","Type":"ContainerDied","Data":"9ab17cfc4658a2cfb9d9f9b730f8c1a1c4f918da589aeae241f07cae93644cd9"} Oct 31 01:03:42.191073 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:42.191008 4440 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ab17cfc4658a2cfb9d9f9b730f8c1a1c4f918da589aeae241f07cae93644cd9" Oct 31 01:03:42.191073 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:42.191029 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wb9gd" Oct 31 01:03:42.475260 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:42.475158 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:42.591555 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:42.591525 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:43.105978 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.105946 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:43.264709 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.264679 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7f58f8c74c-gl94n"] Oct 31 01:03:43.265065 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.264968 4440 cpu_manager.go:414] "containerMap: removing container" podUID="fffeb29e-3429-4a1e-a589-11367e62c490" containerName="util" Oct 31 01:03:43.265065 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.264978 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fffeb29e-3429-4a1e-a589-11367e62c490" containerName="util" Oct 31 01:03:43.265065 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.264989 4440 cpu_manager.go:414] "containerMap: removing container" podUID="3da0ab7f-932f-4367-a609-fe25d23cac66" containerName="extract" Oct 31 01:03:43.265065 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.264995 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3da0ab7f-932f-4367-a609-fe25d23cac66" containerName="extract" Oct 31 01:03:43.265065 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265007 4440 cpu_manager.go:414] "containerMap: removing container" podUID="c2ea521c-7769-4b06-9511-c2a63c2e15aa" containerName="util" Oct 31 01:03:43.265065 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265012 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2ea521c-7769-4b06-9511-c2a63c2e15aa" containerName="util" Oct 31 01:03:43.265065 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265018 4440 cpu_manager.go:414] "containerMap: removing container" podUID="6193fb88-ac70-40dc-bb5d-e37b6e821959" containerName="pull" Oct 31 01:03:43.265065 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265022 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6193fb88-ac70-40dc-bb5d-e37b6e821959" containerName="pull" Oct 31 01:03:43.265065 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265031 4440 cpu_manager.go:414] "containerMap: removing container" podUID="885757ee-5daa-4621-a6b8-68cf24a7b4c5" containerName="pull" Oct 31 01:03:43.265065 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265035 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="885757ee-5daa-4621-a6b8-68cf24a7b4c5" containerName="pull" Oct 31 01:03:43.265065 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265042 4440 cpu_manager.go:414] "containerMap: removing container" podUID="489ce419-a3c4-4de9-b2a4-4fdaff265be9" containerName="pull" Oct 31 01:03:43.265065 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265047 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="489ce419-a3c4-4de9-b2a4-4fdaff265be9" containerName="pull" Oct 31 01:03:43.265065 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265053 4440 cpu_manager.go:414] "containerMap: removing container" podUID="885757ee-5daa-4621-a6b8-68cf24a7b4c5" containerName="extract" Oct 31 01:03:43.265065 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265058 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="885757ee-5daa-4621-a6b8-68cf24a7b4c5" containerName="extract" Oct 31 01:03:43.265065 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265064 4440 cpu_manager.go:414] "containerMap: removing container" podUID="489ce419-a3c4-4de9-b2a4-4fdaff265be9" containerName="extract" Oct 31 01:03:43.265065 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265069 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="489ce419-a3c4-4de9-b2a4-4fdaff265be9" containerName="extract" Oct 31 01:03:43.265065 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265075 4440 cpu_manager.go:414] "containerMap: removing container" podUID="e05eb907-6ddc-4767-9d8b-aad5f0ed41e1" containerName="util" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265080 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e05eb907-6ddc-4767-9d8b-aad5f0ed41e1" containerName="util" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265085 4440 cpu_manager.go:414] "containerMap: removing container" podUID="6193fb88-ac70-40dc-bb5d-e37b6e821959" containerName="util" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265089 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6193fb88-ac70-40dc-bb5d-e37b6e821959" containerName="util" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265095 4440 cpu_manager.go:414] "containerMap: removing container" podUID="9eb24e27-6ef1-44f1-bde2-130f56377c8c" containerName="pull" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265099 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eb24e27-6ef1-44f1-bde2-130f56377c8c" containerName="pull" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265106 4440 cpu_manager.go:414] "containerMap: removing container" podUID="9eb24e27-6ef1-44f1-bde2-130f56377c8c" containerName="util" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265110 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eb24e27-6ef1-44f1-bde2-130f56377c8c" containerName="util" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265116 4440 cpu_manager.go:414] "containerMap: removing container" podUID="e05eb907-6ddc-4767-9d8b-aad5f0ed41e1" containerName="pull" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265120 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e05eb907-6ddc-4767-9d8b-aad5f0ed41e1" containerName="pull" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265127 4440 cpu_manager.go:414] "containerMap: removing container" podUID="7c80a8af-321f-452c-8ee6-cabf283eaf31" containerName="util" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265131 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c80a8af-321f-452c-8ee6-cabf283eaf31" containerName="util" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265136 4440 cpu_manager.go:414] "containerMap: removing container" podUID="7c80a8af-321f-452c-8ee6-cabf283eaf31" containerName="pull" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265140 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c80a8af-321f-452c-8ee6-cabf283eaf31" containerName="pull" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265145 4440 cpu_manager.go:414] "containerMap: removing container" podUID="fffeb29e-3429-4a1e-a589-11367e62c490" containerName="extract" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265149 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fffeb29e-3429-4a1e-a589-11367e62c490" containerName="extract" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265156 4440 cpu_manager.go:414] "containerMap: removing container" podUID="3da0ab7f-932f-4367-a609-fe25d23cac66" containerName="util" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265160 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3da0ab7f-932f-4367-a609-fe25d23cac66" containerName="util" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265165 4440 cpu_manager.go:414] "containerMap: removing container" podUID="489ce419-a3c4-4de9-b2a4-4fdaff265be9" containerName="util" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265170 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="489ce419-a3c4-4de9-b2a4-4fdaff265be9" containerName="util" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265176 4440 cpu_manager.go:414] "containerMap: removing container" podUID="38588f65-5d75-4b58-a7f6-912cc02c2064" containerName="pull" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265180 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38588f65-5d75-4b58-a7f6-912cc02c2064" containerName="pull" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265185 4440 cpu_manager.go:414] "containerMap: removing container" podUID="c2ea521c-7769-4b06-9511-c2a63c2e15aa" containerName="extract" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265189 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2ea521c-7769-4b06-9511-c2a63c2e15aa" containerName="extract" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265196 4440 cpu_manager.go:414] "containerMap: removing container" podUID="e05eb907-6ddc-4767-9d8b-aad5f0ed41e1" containerName="extract" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265201 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e05eb907-6ddc-4767-9d8b-aad5f0ed41e1" containerName="extract" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265209 4440 cpu_manager.go:414] "containerMap: removing container" podUID="885757ee-5daa-4621-a6b8-68cf24a7b4c5" containerName="util" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265213 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="885757ee-5daa-4621-a6b8-68cf24a7b4c5" containerName="util" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265218 4440 cpu_manager.go:414] "containerMap: removing container" podUID="bb3f2b0e-88c7-4792-85ff-02c54f46cafd" containerName="extract" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265222 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb3f2b0e-88c7-4792-85ff-02c54f46cafd" containerName="extract" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265227 4440 cpu_manager.go:414] "containerMap: removing container" podUID="38588f65-5d75-4b58-a7f6-912cc02c2064" containerName="util" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265232 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38588f65-5d75-4b58-a7f6-912cc02c2064" containerName="util" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265237 4440 cpu_manager.go:414] "containerMap: removing container" podUID="d5148e11-ffda-4e2f-be35-f17c13e0515b" containerName="extract" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265242 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5148e11-ffda-4e2f-be35-f17c13e0515b" containerName="extract" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265248 4440 cpu_manager.go:414] "containerMap: removing container" podUID="7c80a8af-321f-452c-8ee6-cabf283eaf31" containerName="extract" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265253 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c80a8af-321f-452c-8ee6-cabf283eaf31" containerName="extract" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265259 4440 cpu_manager.go:414] "containerMap: removing container" podUID="6193fb88-ac70-40dc-bb5d-e37b6e821959" containerName="extract" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265263 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6193fb88-ac70-40dc-bb5d-e37b6e821959" containerName="extract" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265269 4440 cpu_manager.go:414] "containerMap: removing container" podUID="9eb24e27-6ef1-44f1-bde2-130f56377c8c" containerName="extract" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265273 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eb24e27-6ef1-44f1-bde2-130f56377c8c" containerName="extract" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265281 4440 cpu_manager.go:414] "containerMap: removing container" podUID="d5148e11-ffda-4e2f-be35-f17c13e0515b" containerName="util" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265285 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5148e11-ffda-4e2f-be35-f17c13e0515b" containerName="util" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265291 4440 cpu_manager.go:414] "containerMap: removing container" podUID="3da0ab7f-932f-4367-a609-fe25d23cac66" containerName="pull" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265294 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3da0ab7f-932f-4367-a609-fe25d23cac66" containerName="pull" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265300 4440 cpu_manager.go:414] "containerMap: removing container" podUID="d5148e11-ffda-4e2f-be35-f17c13e0515b" containerName="pull" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265304 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5148e11-ffda-4e2f-be35-f17c13e0515b" containerName="pull" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265310 4440 cpu_manager.go:414] "containerMap: removing container" podUID="bb3f2b0e-88c7-4792-85ff-02c54f46cafd" containerName="util" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265314 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb3f2b0e-88c7-4792-85ff-02c54f46cafd" containerName="util" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265319 4440 cpu_manager.go:414] "containerMap: removing container" podUID="c2ea521c-7769-4b06-9511-c2a63c2e15aa" containerName="pull" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265323 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2ea521c-7769-4b06-9511-c2a63c2e15aa" containerName="pull" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265329 4440 cpu_manager.go:414] "containerMap: removing container" podUID="fffeb29e-3429-4a1e-a589-11367e62c490" containerName="pull" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265333 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fffeb29e-3429-4a1e-a589-11367e62c490" containerName="pull" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265337 4440 cpu_manager.go:414] "containerMap: removing container" podUID="bb3f2b0e-88c7-4792-85ff-02c54f46cafd" containerName="pull" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265342 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb3f2b0e-88c7-4792-85ff-02c54f46cafd" containerName="pull" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265348 4440 cpu_manager.go:414] "containerMap: removing container" podUID="38588f65-5d75-4b58-a7f6-912cc02c2064" containerName="extract" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265352 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38588f65-5d75-4b58-a7f6-912cc02c2064" containerName="extract" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265390 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="fffeb29e-3429-4a1e-a589-11367e62c490" containerName="extract" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265397 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2ea521c-7769-4b06-9511-c2a63c2e15aa" containerName="extract" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265404 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="e05eb907-6ddc-4767-9d8b-aad5f0ed41e1" containerName="extract" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265408 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb3f2b0e-88c7-4792-85ff-02c54f46cafd" containerName="extract" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265413 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5148e11-ffda-4e2f-be35-f17c13e0515b" containerName="extract" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265432 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c80a8af-321f-452c-8ee6-cabf283eaf31" containerName="extract" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265437 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="6193fb88-ac70-40dc-bb5d-e37b6e821959" containerName="extract" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265442 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eb24e27-6ef1-44f1-bde2-130f56377c8c" containerName="extract" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265449 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="38588f65-5d75-4b58-a7f6-912cc02c2064" containerName="extract" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265454 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="3da0ab7f-932f-4367-a609-fe25d23cac66" containerName="extract" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265460 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="489ce419-a3c4-4de9-b2a4-4fdaff265be9" containerName="extract" Oct 31 01:03:43.265595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.265465 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="885757ee-5daa-4621-a6b8-68cf24a7b4c5" containerName="extract" Oct 31 01:03:43.268239 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.268222 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f58f8c74c-gl94n" Oct 31 01:03:43.275143 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.275122 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 31 01:03:43.275742 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.275725 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 31 01:03:43.275881 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.275868 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-tml79\"" Oct 31 01:03:43.275975 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.275950 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 31 01:03:43.275975 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.275954 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 31 01:03:43.276096 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.275988 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 31 01:03:43.301093 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.301070 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7f58f8c74c-gl94n"] Oct 31 01:03:43.338316 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.338278 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/513526cf-d6aa-47fe-a8a8-e2191cf2be3d-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f58f8c74c-gl94n\" (UID: \"513526cf-d6aa-47fe-a8a8-e2191cf2be3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f58f8c74c-gl94n" Oct 31 01:03:43.338539 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.338339 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/513526cf-d6aa-47fe-a8a8-e2191cf2be3d-metrics-cert\") pod \"opendatahub-operator-controller-manager-7f58f8c74c-gl94n\" (UID: \"513526cf-d6aa-47fe-a8a8-e2191cf2be3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f58f8c74c-gl94n" Oct 31 01:03:43.338539 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.338376 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/513526cf-d6aa-47fe-a8a8-e2191cf2be3d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f58f8c74c-gl94n\" (UID: \"513526cf-d6aa-47fe-a8a8-e2191cf2be3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f58f8c74c-gl94n" Oct 31 01:03:43.338539 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.338441 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcckh\" (UniqueName: \"kubernetes.io/projected/513526cf-d6aa-47fe-a8a8-e2191cf2be3d-kube-api-access-gcckh\") pod \"opendatahub-operator-controller-manager-7f58f8c74c-gl94n\" (UID: \"513526cf-d6aa-47fe-a8a8-e2191cf2be3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f58f8c74c-gl94n" Oct 31 01:03:43.338539 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.338517 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/513526cf-d6aa-47fe-a8a8-e2191cf2be3d-manifests\") pod \"opendatahub-operator-controller-manager-7f58f8c74c-gl94n\" (UID: \"513526cf-d6aa-47fe-a8a8-e2191cf2be3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f58f8c74c-gl94n" Oct 31 01:03:43.338745 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.338696 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/513526cf-d6aa-47fe-a8a8-e2191cf2be3d-tmp\") pod \"opendatahub-operator-controller-manager-7f58f8c74c-gl94n\" (UID: \"513526cf-d6aa-47fe-a8a8-e2191cf2be3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f58f8c74c-gl94n" Oct 31 01:03:43.439309 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.439277 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/513526cf-d6aa-47fe-a8a8-e2191cf2be3d-tmp\") pod \"opendatahub-operator-controller-manager-7f58f8c74c-gl94n\" (UID: \"513526cf-d6aa-47fe-a8a8-e2191cf2be3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f58f8c74c-gl94n" Oct 31 01:03:43.439517 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.439324 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/513526cf-d6aa-47fe-a8a8-e2191cf2be3d-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f58f8c74c-gl94n\" (UID: \"513526cf-d6aa-47fe-a8a8-e2191cf2be3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f58f8c74c-gl94n" Oct 31 01:03:43.439517 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.439347 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/513526cf-d6aa-47fe-a8a8-e2191cf2be3d-metrics-cert\") pod \"opendatahub-operator-controller-manager-7f58f8c74c-gl94n\" (UID: \"513526cf-d6aa-47fe-a8a8-e2191cf2be3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f58f8c74c-gl94n" Oct 31 01:03:43.439517 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.439373 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/513526cf-d6aa-47fe-a8a8-e2191cf2be3d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f58f8c74c-gl94n\" (UID: \"513526cf-d6aa-47fe-a8a8-e2191cf2be3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f58f8c74c-gl94n" Oct 31 01:03:43.439517 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.439412 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcckh\" (UniqueName: \"kubernetes.io/projected/513526cf-d6aa-47fe-a8a8-e2191cf2be3d-kube-api-access-gcckh\") pod \"opendatahub-operator-controller-manager-7f58f8c74c-gl94n\" (UID: \"513526cf-d6aa-47fe-a8a8-e2191cf2be3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f58f8c74c-gl94n" Oct 31 01:03:43.439517 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.439451 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/513526cf-d6aa-47fe-a8a8-e2191cf2be3d-manifests\") pod \"opendatahub-operator-controller-manager-7f58f8c74c-gl94n\" (UID: \"513526cf-d6aa-47fe-a8a8-e2191cf2be3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f58f8c74c-gl94n" Oct 31 01:03:43.439830 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.439804 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/513526cf-d6aa-47fe-a8a8-e2191cf2be3d-manifests\") pod \"opendatahub-operator-controller-manager-7f58f8c74c-gl94n\" (UID: \"513526cf-d6aa-47fe-a8a8-e2191cf2be3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f58f8c74c-gl94n" Oct 31 01:03:43.441824 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.441788 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/513526cf-d6aa-47fe-a8a8-e2191cf2be3d-tmp\") pod \"opendatahub-operator-controller-manager-7f58f8c74c-gl94n\" (UID: \"513526cf-d6aa-47fe-a8a8-e2191cf2be3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f58f8c74c-gl94n" Oct 31 01:03:43.441944 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.441837 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/513526cf-d6aa-47fe-a8a8-e2191cf2be3d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f58f8c74c-gl94n\" (UID: \"513526cf-d6aa-47fe-a8a8-e2191cf2be3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f58f8c74c-gl94n" Oct 31 01:03:43.441944 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.441916 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/513526cf-d6aa-47fe-a8a8-e2191cf2be3d-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f58f8c74c-gl94n\" (UID: \"513526cf-d6aa-47fe-a8a8-e2191cf2be3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f58f8c74c-gl94n" Oct 31 01:03:43.442037 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.441939 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/513526cf-d6aa-47fe-a8a8-e2191cf2be3d-metrics-cert\") pod \"opendatahub-operator-controller-manager-7f58f8c74c-gl94n\" (UID: \"513526cf-d6aa-47fe-a8a8-e2191cf2be3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f58f8c74c-gl94n" Oct 31 01:03:43.448226 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.448201 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcckh\" (UniqueName: \"kubernetes.io/projected/513526cf-d6aa-47fe-a8a8-e2191cf2be3d-kube-api-access-gcckh\") pod \"opendatahub-operator-controller-manager-7f58f8c74c-gl94n\" (UID: \"513526cf-d6aa-47fe-a8a8-e2191cf2be3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f58f8c74c-gl94n" Oct 31 01:03:43.476970 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.476939 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:43.581233 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.581206 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f58f8c74c-gl94n" Oct 31 01:03:43.602833 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.602804 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:43.740320 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.740232 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:43.869230 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.869190 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:43.882138 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:43.882117 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7f58f8c74c-gl94n"] Oct 31 01:03:43.883903 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:03:43.883869 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod513526cf_d6aa_47fe_a8a8_e2191cf2be3d.slice/crio-b004622986df4948732b2703ac4e208ea3ed44981ecbbf25fc4bf9887e536c9c WatchSource:0}: Error finding container b004622986df4948732b2703ac4e208ea3ed44981ecbbf25fc4bf9887e536c9c: Status 404 returned error can't find the container with id b004622986df4948732b2703ac4e208ea3ed44981ecbbf25fc4bf9887e536c9c Oct 31 01:03:44.202015 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:44.201983 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f58f8c74c-gl94n" event={"ID":"513526cf-d6aa-47fe-a8a8-e2191cf2be3d","Type":"ContainerStarted","Data":"b004622986df4948732b2703ac4e208ea3ed44981ecbbf25fc4bf9887e536c9c"} Oct 31 01:03:44.308723 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:44.308687 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:45.426156 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:45.426116 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:46.490549 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:46.490512 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:46.608111 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:46.608077 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:46.730585 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:46.730539 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:47.218017 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:47.217979 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f58f8c74c-gl94n" event={"ID":"513526cf-d6aa-47fe-a8a8-e2191cf2be3d","Type":"ContainerStarted","Data":"8aae0abd802232242e12381b2fed1669c01f5abf91fcb735af3cfcbbe98aa062"} Oct 31 01:03:48.224673 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:48.224639 4440 generic.go:356] "Generic (PLEG): container finished" podID="513526cf-d6aa-47fe-a8a8-e2191cf2be3d" containerID="8aae0abd802232242e12381b2fed1669c01f5abf91fcb735af3cfcbbe98aa062" exitCode=0 Oct 31 01:03:48.225137 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:48.224711 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f58f8c74c-gl94n" event={"ID":"513526cf-d6aa-47fe-a8a8-e2191cf2be3d","Type":"ContainerDied","Data":"8aae0abd802232242e12381b2fed1669c01f5abf91fcb735af3cfcbbe98aa062"} Oct 31 01:03:49.231160 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:49.231104 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f58f8c74c-gl94n" event={"ID":"513526cf-d6aa-47fe-a8a8-e2191cf2be3d","Type":"ContainerStarted","Data":"e77b37d0b29f233c30e838f4ee4c5ef3f714ec8e5ec966aa97094b892c5ff8cc"} Oct 31 01:03:49.231703 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:49.231250 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f58f8c74c-gl94n" Oct 31 01:03:49.276610 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:49.276547 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f58f8c74c-gl94n" podStartSLOduration=2.225366201 podStartE2EDuration="6.276531836s" totalImagesPullingTime="3.621244204s" totalInitContainerRuntime="429.921431ms" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:49.273394025 +0000 UTC m=+473.632116341" watchObservedRunningTime="2026-10-31 01:03:49.276531836 +0000 UTC m=+473.635254152" Oct 31 01:03:51.882047 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:51.882015 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:52.020494 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:52.020414 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:55.250205 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:55.250167 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:56.247196 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:56.247169 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:58.038935 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:03:58.038897 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:00.238332 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:00.238294 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f58f8c74c-gl94n" Oct 31 01:04:00.609496 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:00.609377 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-tmjs6"] Oct 31 01:04:00.619710 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:00.619676 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-tmjs6" Oct 31 01:04:00.624385 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:00.624357 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-5g7mv\"" Oct 31 01:04:00.625036 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:00.625012 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 01:04:00.625239 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:00.625224 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 01:04:00.627371 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:00.627348 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-tmjs6"] Oct 31 01:04:00.673412 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:00.673373 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfp5k\" (UniqueName: \"kubernetes.io/projected/e7fa1604-b8a5-4dc7-9b70-2f0eacebaf76-kube-api-access-kfp5k\") pod \"authorino-operator-7b7bb4bc77-tmjs6\" (UID: \"e7fa1604-b8a5-4dc7-9b70-2f0eacebaf76\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-tmjs6" Oct 31 01:04:00.774295 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:00.774250 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfp5k\" (UniqueName: \"kubernetes.io/projected/e7fa1604-b8a5-4dc7-9b70-2f0eacebaf76-kube-api-access-kfp5k\") pod \"authorino-operator-7b7bb4bc77-tmjs6\" (UID: \"e7fa1604-b8a5-4dc7-9b70-2f0eacebaf76\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-tmjs6" Oct 31 01:04:00.786360 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:00.786317 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfp5k\" (UniqueName: \"kubernetes.io/projected/e7fa1604-b8a5-4dc7-9b70-2f0eacebaf76-kube-api-access-kfp5k\") pod \"authorino-operator-7b7bb4bc77-tmjs6\" (UID: \"e7fa1604-b8a5-4dc7-9b70-2f0eacebaf76\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-tmjs6" Oct 31 01:04:00.934566 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:00.934526 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-tmjs6" Oct 31 01:04:01.024608 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:04:01.024568 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:04:01.223603 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:01.223576 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-tmjs6"] Oct 31 01:04:01.225639 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:04:01.225609 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7fa1604_b8a5_4dc7_9b70_2f0eacebaf76.slice/crio-c97a55dbb5df00facbb27dbe09ffacc6ccb8d0feff8014b749b7d21a645d8637 WatchSource:0}: Error finding container c97a55dbb5df00facbb27dbe09ffacc6ccb8d0feff8014b749b7d21a645d8637: Status 404 returned error can't find the container with id c97a55dbb5df00facbb27dbe09ffacc6ccb8d0feff8014b749b7d21a645d8637 Oct 31 01:04:01.287018 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:01.286983 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-tmjs6" event={"ID":"e7fa1604-b8a5-4dc7-9b70-2f0eacebaf76","Type":"ContainerStarted","Data":"c97a55dbb5df00facbb27dbe09ffacc6ccb8d0feff8014b749b7d21a645d8637"} Oct 31 01:04:02.068154 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:02.068119 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:02.264970 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:02.264935 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7tsvv"] Oct 31 01:04:02.299747 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:02.299719 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7tsvv"] Oct 31 01:04:02.300187 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:02.299814 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7tsvv" Oct 31 01:04:02.315455 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:02.315409 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-rftns\"" Oct 31 01:04:02.315583 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:02.315471 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 31 01:04:02.315583 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:02.315483 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 31 01:04:02.387156 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:02.387068 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b77f8c77-df9a-4107-925f-8b48c19a0b31-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-7tsvv\" (UID: \"b77f8c77-df9a-4107-925f-8b48c19a0b31\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7tsvv" Oct 31 01:04:02.387321 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:02.387200 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skwr4\" (UniqueName: \"kubernetes.io/projected/b77f8c77-df9a-4107-925f-8b48c19a0b31-kube-api-access-skwr4\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-7tsvv\" (UID: \"b77f8c77-df9a-4107-925f-8b48c19a0b31\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7tsvv" Oct 31 01:04:02.488241 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:02.488206 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skwr4\" (UniqueName: \"kubernetes.io/projected/b77f8c77-df9a-4107-925f-8b48c19a0b31-kube-api-access-skwr4\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-7tsvv\" (UID: \"b77f8c77-df9a-4107-925f-8b48c19a0b31\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7tsvv" Oct 31 01:04:02.488241 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:02.488245 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b77f8c77-df9a-4107-925f-8b48c19a0b31-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-7tsvv\" (UID: \"b77f8c77-df9a-4107-925f-8b48c19a0b31\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7tsvv" Oct 31 01:04:02.488647 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:02.488629 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b77f8c77-df9a-4107-925f-8b48c19a0b31-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-7tsvv\" (UID: \"b77f8c77-df9a-4107-925f-8b48c19a0b31\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7tsvv" Oct 31 01:04:02.498231 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:02.498208 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skwr4\" (UniqueName: \"kubernetes.io/projected/b77f8c77-df9a-4107-925f-8b48c19a0b31-kube-api-access-skwr4\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-7tsvv\" (UID: \"b77f8c77-df9a-4107-925f-8b48c19a0b31\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7tsvv" Oct 31 01:04:02.626169 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:02.626138 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7tsvv" Oct 31 01:04:02.934583 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:02.934231 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7tsvv"] Oct 31 01:04:02.940106 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:04:02.940072 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb77f8c77_df9a_4107_925f_8b48c19a0b31.slice/crio-ead382c5950ad8d5a2cacd9d8ee5ee6e481fa7b8af8ee5c0280e6a2dffb8f12b WatchSource:0}: Error finding container ead382c5950ad8d5a2cacd9d8ee5ee6e481fa7b8af8ee5c0280e6a2dffb8f12b: Status 404 returned error can't find the container with id ead382c5950ad8d5a2cacd9d8ee5ee6e481fa7b8af8ee5c0280e6a2dffb8f12b Oct 31 01:04:03.307968 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:03.307852 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7tsvv" event={"ID":"b77f8c77-df9a-4107-925f-8b48c19a0b31","Type":"ContainerStarted","Data":"ead382c5950ad8d5a2cacd9d8ee5ee6e481fa7b8af8ee5c0280e6a2dffb8f12b"} Oct 31 01:04:04.467067 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:04.467039 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-4k268"] Oct 31 01:04:04.470255 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:04.470231 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-4k268" Oct 31 01:04:04.478730 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:04.478709 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-f8bg5\"" Oct 31 01:04:04.485102 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:04.485078 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-4k268"] Oct 31 01:04:04.492241 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:04.492218 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 31 01:04:04.505984 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:04.505934 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcb5h\" (UniqueName: \"kubernetes.io/projected/5d812e97-43dd-4bd1-b689-410fa1bc32c4-kube-api-access-xcb5h\") pod \"dns-operator-controller-manager-8557cbdb84-4k268\" (UID: \"5d812e97-43dd-4bd1-b689-410fa1bc32c4\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-4k268" Oct 31 01:04:04.606458 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:04.606394 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcb5h\" (UniqueName: \"kubernetes.io/projected/5d812e97-43dd-4bd1-b689-410fa1bc32c4-kube-api-access-xcb5h\") pod \"dns-operator-controller-manager-8557cbdb84-4k268\" (UID: \"5d812e97-43dd-4bd1-b689-410fa1bc32c4\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-4k268" Oct 31 01:04:04.620657 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:04.620616 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcb5h\" (UniqueName: \"kubernetes.io/projected/5d812e97-43dd-4bd1-b689-410fa1bc32c4-kube-api-access-xcb5h\") pod \"dns-operator-controller-manager-8557cbdb84-4k268\" (UID: \"5d812e97-43dd-4bd1-b689-410fa1bc32c4\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-4k268" Oct 31 01:04:04.782550 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:04.782503 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-4k268" Oct 31 01:04:05.062765 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:05.062738 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-4k268"] Oct 31 01:04:05.064818 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:04:05.064784 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d812e97_43dd_4bd1_b689_410fa1bc32c4.slice/crio-923e2f8bb6720c8517b51270fcb2b041a670d18579a3d5d4ccafe238b54358ab WatchSource:0}: Error finding container 923e2f8bb6720c8517b51270fcb2b041a670d18579a3d5d4ccafe238b54358ab: Status 404 returned error can't find the container with id 923e2f8bb6720c8517b51270fcb2b041a670d18579a3d5d4ccafe238b54358ab Oct 31 01:04:05.319737 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:05.319655 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-tmjs6" event={"ID":"e7fa1604-b8a5-4dc7-9b70-2f0eacebaf76","Type":"ContainerStarted","Data":"5e84c7c6e1f9bf09083a6e3f9607f852508f4170a683b00fe89208b64e983b5b"} Oct 31 01:04:05.319737 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:05.319726 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-tmjs6" Oct 31 01:04:05.321274 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:05.321248 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-4k268" event={"ID":"5d812e97-43dd-4bd1-b689-410fa1bc32c4","Type":"ContainerStarted","Data":"923e2f8bb6720c8517b51270fcb2b041a670d18579a3d5d4ccafe238b54358ab"} Oct 31 01:04:05.340386 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:05.340333 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-tmjs6" podStartSLOduration=2.151401234 podStartE2EDuration="5.340317629s" totalImagesPullingTime="3.188916395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:05.337633556 +0000 UTC m=+489.696355871" watchObservedRunningTime="2026-10-31 01:04:05.340317629 +0000 UTC m=+489.699039967" Oct 31 01:04:05.999885 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:05.999779 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:06.328564 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:06.328461 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7tsvv" event={"ID":"b77f8c77-df9a-4107-925f-8b48c19a0b31","Type":"ContainerStarted","Data":"a119741b59940fedd76e28dd3500664cd6b8cd1cd1b297dec9f299a1feb8decf"} Oct 31 01:04:06.380096 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:06.380030 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7tsvv" podStartSLOduration=1.6358959849999999 podStartE2EDuration="4.38001085s" totalImagesPullingTime="2.744114865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:06.378038369 +0000 UTC m=+490.736760699" watchObservedRunningTime="2026-10-31 01:04:06.38001085 +0000 UTC m=+490.738733168" Oct 31 01:04:06.799632 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:06.799586 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:07.013028 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:07.012922 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:08.339668 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:08.339632 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-4k268" event={"ID":"5d812e97-43dd-4bd1-b689-410fa1bc32c4","Type":"ContainerStarted","Data":"24c56b8861cf2ed0cbedcf7c7f7aef6a02cbe47957c03f7e0c01521f092ef531"} Oct 31 01:04:08.340059 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:08.339744 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-4k268" Oct 31 01:04:08.365752 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:08.365696 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-4k268" podStartSLOduration=1.535516063 podStartE2EDuration="4.365680144s" totalImagesPullingTime="2.830164081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:08.36449 +0000 UTC m=+492.723212317" watchObservedRunningTime="2026-10-31 01:04:08.365680144 +0000 UTC m=+492.724402459" Oct 31 01:04:08.391828 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:08.391788 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:08.625871 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:08.625774 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-qw7v5"] Oct 31 01:04:08.629231 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:08.629206 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-qw7v5" Oct 31 01:04:08.648903 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:08.648875 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-wqvwf\"" Oct 31 01:04:08.650492 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:08.650467 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-qw7v5"] Oct 31 01:04:08.741497 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:08.741448 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87ks2\" (UniqueName: \"kubernetes.io/projected/4d6ca668-5b11-4b95-a803-00694d4522e2-kube-api-access-87ks2\") pod \"limitador-operator-controller-manager-5778bb4449-qw7v5\" (UID: \"4d6ca668-5b11-4b95-a803-00694d4522e2\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-qw7v5" Oct 31 01:04:08.842312 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:08.842264 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87ks2\" (UniqueName: \"kubernetes.io/projected/4d6ca668-5b11-4b95-a803-00694d4522e2-kube-api-access-87ks2\") pod \"limitador-operator-controller-manager-5778bb4449-qw7v5\" (UID: \"4d6ca668-5b11-4b95-a803-00694d4522e2\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-qw7v5" Oct 31 01:04:08.861679 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:08.861650 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87ks2\" (UniqueName: \"kubernetes.io/projected/4d6ca668-5b11-4b95-a803-00694d4522e2-kube-api-access-87ks2\") pod \"limitador-operator-controller-manager-5778bb4449-qw7v5\" (UID: \"4d6ca668-5b11-4b95-a803-00694d4522e2\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-qw7v5" Oct 31 01:04:08.938942 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:08.938898 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:08.941737 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:08.941720 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-qw7v5" Oct 31 01:04:09.138478 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:09.138442 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:09.219556 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:09.219531 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-qw7v5"] Oct 31 01:04:09.222005 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:04:09.221968 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d6ca668_5b11_4b95_a803_00694d4522e2.slice/crio-25d6f1d9cf018329a5242ee0823476d807a438a23fa56638cb5eb58079f05cd2 WatchSource:0}: Error finding container 25d6f1d9cf018329a5242ee0823476d807a438a23fa56638cb5eb58079f05cd2: Status 404 returned error can't find the container with id 25d6f1d9cf018329a5242ee0823476d807a438a23fa56638cb5eb58079f05cd2 Oct 31 01:04:09.346608 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:09.346569 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-qw7v5" event={"ID":"4d6ca668-5b11-4b95-a803-00694d4522e2","Type":"ContainerStarted","Data":"25d6f1d9cf018329a5242ee0823476d807a438a23fa56638cb5eb58079f05cd2"} Oct 31 01:04:09.358325 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:09.358290 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:09.655646 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:09.655548 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:10.469844 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:10.469814 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:11.496683 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:11.496644 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:11.690194 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:11.690153 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:12.362961 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:12.362921 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-qw7v5" event={"ID":"4d6ca668-5b11-4b95-a803-00694d4522e2","Type":"ContainerStarted","Data":"11ef0a0a6f06baf20e8e6d80bc11d7d18986d4ab9bc48ba121094126689e9546"} Oct 31 01:04:12.363181 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:12.363043 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-qw7v5" Oct 31 01:04:12.381382 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:12.381326 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-qw7v5" podStartSLOduration=1.797783875 podStartE2EDuration="4.381311146s" totalImagesPullingTime="2.583527271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:12.380348857 +0000 UTC m=+496.739071184" watchObservedRunningTime="2026-10-31 01:04:12.381311146 +0000 UTC m=+496.740033461" Oct 31 01:04:15.032545 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:15.032512 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:15.495652 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:15.495613 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5thg5"] Oct 31 01:04:15.500143 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:15.500124 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5thg5" Oct 31 01:04:15.509568 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:15.509541 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-cblcn\"" Oct 31 01:04:15.517031 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:15.517000 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5thg5"] Oct 31 01:04:15.596739 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:15.596702 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9166216f-113a-4d6a-972e-1d0fc5d36ebb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-5thg5\" (UID: \"9166216f-113a-4d6a-972e-1d0fc5d36ebb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5thg5" Oct 31 01:04:15.596953 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:15.596789 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qnfv\" (UniqueName: \"kubernetes.io/projected/9166216f-113a-4d6a-972e-1d0fc5d36ebb-kube-api-access-9qnfv\") pod \"kuadrant-operator-controller-manager-cf8469d7b-5thg5\" (UID: \"9166216f-113a-4d6a-972e-1d0fc5d36ebb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5thg5" Oct 31 01:04:15.698221 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:15.698181 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qnfv\" (UniqueName: \"kubernetes.io/projected/9166216f-113a-4d6a-972e-1d0fc5d36ebb-kube-api-access-9qnfv\") pod \"kuadrant-operator-controller-manager-cf8469d7b-5thg5\" (UID: \"9166216f-113a-4d6a-972e-1d0fc5d36ebb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5thg5" Oct 31 01:04:15.698221 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:15.698223 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9166216f-113a-4d6a-972e-1d0fc5d36ebb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-5thg5\" (UID: \"9166216f-113a-4d6a-972e-1d0fc5d36ebb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5thg5" Oct 31 01:04:15.698647 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:15.698630 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9166216f-113a-4d6a-972e-1d0fc5d36ebb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-5thg5\" (UID: \"9166216f-113a-4d6a-972e-1d0fc5d36ebb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5thg5" Oct 31 01:04:15.710682 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:15.710659 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qnfv\" (UniqueName: \"kubernetes.io/projected/9166216f-113a-4d6a-972e-1d0fc5d36ebb-kube-api-access-9qnfv\") pod \"kuadrant-operator-controller-manager-cf8469d7b-5thg5\" (UID: \"9166216f-113a-4d6a-972e-1d0fc5d36ebb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5thg5" Oct 31 01:04:15.812363 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:15.812296 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5thg5" Oct 31 01:04:16.092777 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:16.092700 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5thg5"] Oct 31 01:04:16.096240 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:04:16.096216 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9166216f_113a_4d6a_972e_1d0fc5d36ebb.slice/crio-76c9b65db042d37378b7c1de97386e97e662b150b867aeaf6a143dc77e4e7907 WatchSource:0}: Error finding container 76c9b65db042d37378b7c1de97386e97e662b150b867aeaf6a143dc77e4e7907: Status 404 returned error can't find the container with id 76c9b65db042d37378b7c1de97386e97e662b150b867aeaf6a143dc77e4e7907 Oct 31 01:04:16.331315 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:16.331271 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-tmjs6" Oct 31 01:04:16.384524 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:16.384434 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5thg5" event={"ID":"9166216f-113a-4d6a-972e-1d0fc5d36ebb","Type":"ContainerStarted","Data":"76c9b65db042d37378b7c1de97386e97e662b150b867aeaf6a143dc77e4e7907"} Oct 31 01:04:16.547663 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:16.547620 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:19.349795 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:19.349766 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-4k268" Oct 31 01:04:20.637054 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:20.637013 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:20.776543 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:20.776509 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:20.929270 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:20.929243 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:21.012323 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:21.012286 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557t8mkr"] Oct 31 01:04:21.018770 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:21.018745 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557t8mkr" Oct 31 01:04:21.028090 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:21.028066 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-lb9xp\"" Oct 31 01:04:21.036797 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:21.036776 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557t8mkr"] Oct 31 01:04:21.104230 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:21.104137 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:21.145713 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:21.145680 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c5e03e2-611d-4ef6-804d-a769a4148613-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68d996c557t8mkr\" (UID: \"0c5e03e2-611d-4ef6-804d-a769a4148613\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557t8mkr" Oct 31 01:04:21.145890 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:21.145720 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c5e03e2-611d-4ef6-804d-a769a4148613-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68d996c557t8mkr\" (UID: \"0c5e03e2-611d-4ef6-804d-a769a4148613\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557t8mkr" Oct 31 01:04:21.145890 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:21.145754 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c5e03e2-611d-4ef6-804d-a769a4148613-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68d996c557t8mkr\" (UID: \"0c5e03e2-611d-4ef6-804d-a769a4148613\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557t8mkr" Oct 31 01:04:21.146010 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:21.145911 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c5e03e2-611d-4ef6-804d-a769a4148613-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68d996c557t8mkr\" (UID: \"0c5e03e2-611d-4ef6-804d-a769a4148613\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557t8mkr" Oct 31 01:04:21.146010 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:21.145972 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c5e03e2-611d-4ef6-804d-a769a4148613-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68d996c557t8mkr\" (UID: \"0c5e03e2-611d-4ef6-804d-a769a4148613\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557t8mkr" Oct 31 01:04:21.146116 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:21.146010 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsntg\" (UniqueName: \"kubernetes.io/projected/0c5e03e2-611d-4ef6-804d-a769a4148613-kube-api-access-jsntg\") pod \"data-science-gateway-data-science-gateway-class-68d996c557t8mkr\" (UID: \"0c5e03e2-611d-4ef6-804d-a769a4148613\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557t8mkr" Oct 31 01:04:21.146116 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:21.146041 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c5e03e2-611d-4ef6-804d-a769a4148613-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68d996c557t8mkr\" (UID: \"0c5e03e2-611d-4ef6-804d-a769a4148613\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557t8mkr" Oct 31 01:04:21.146116 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:21.146068 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c5e03e2-611d-4ef6-804d-a769a4148613-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68d996c557t8mkr\" (UID: \"0c5e03e2-611d-4ef6-804d-a769a4148613\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557t8mkr" Oct 31 01:04:21.146213 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:21.146146 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c5e03e2-611d-4ef6-804d-a769a4148613-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68d996c557t8mkr\" (UID: \"0c5e03e2-611d-4ef6-804d-a769a4148613\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557t8mkr" Oct 31 01:04:21.247484 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:21.247395 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c5e03e2-611d-4ef6-804d-a769a4148613-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68d996c557t8mkr\" (UID: \"0c5e03e2-611d-4ef6-804d-a769a4148613\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557t8mkr" Oct 31 01:04:21.247484 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:21.247450 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c5e03e2-611d-4ef6-804d-a769a4148613-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68d996c557t8mkr\" (UID: \"0c5e03e2-611d-4ef6-804d-a769a4148613\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557t8mkr" Oct 31 01:04:21.247484 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:21.247484 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c5e03e2-611d-4ef6-804d-a769a4148613-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68d996c557t8mkr\" (UID: \"0c5e03e2-611d-4ef6-804d-a769a4148613\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557t8mkr" Oct 31 01:04:21.247777 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:21.247530 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c5e03e2-611d-4ef6-804d-a769a4148613-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68d996c557t8mkr\" (UID: \"0c5e03e2-611d-4ef6-804d-a769a4148613\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557t8mkr" Oct 31 01:04:21.247777 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:21.247694 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c5e03e2-611d-4ef6-804d-a769a4148613-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68d996c557t8mkr\" (UID: \"0c5e03e2-611d-4ef6-804d-a769a4148613\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557t8mkr" Oct 31 01:04:21.247777 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:21.247743 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsntg\" (UniqueName: \"kubernetes.io/projected/0c5e03e2-611d-4ef6-804d-a769a4148613-kube-api-access-jsntg\") pod \"data-science-gateway-data-science-gateway-class-68d996c557t8mkr\" (UID: \"0c5e03e2-611d-4ef6-804d-a769a4148613\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557t8mkr" Oct 31 01:04:21.247939 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:21.247776 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c5e03e2-611d-4ef6-804d-a769a4148613-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68d996c557t8mkr\" (UID: \"0c5e03e2-611d-4ef6-804d-a769a4148613\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557t8mkr" Oct 31 01:04:21.247939 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:21.247802 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c5e03e2-611d-4ef6-804d-a769a4148613-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68d996c557t8mkr\" (UID: \"0c5e03e2-611d-4ef6-804d-a769a4148613\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557t8mkr" Oct 31 01:04:21.247939 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:21.247835 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c5e03e2-611d-4ef6-804d-a769a4148613-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68d996c557t8mkr\" (UID: \"0c5e03e2-611d-4ef6-804d-a769a4148613\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557t8mkr" Oct 31 01:04:21.247939 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:21.247873 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c5e03e2-611d-4ef6-804d-a769a4148613-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68d996c557t8mkr\" (UID: \"0c5e03e2-611d-4ef6-804d-a769a4148613\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557t8mkr" Oct 31 01:04:21.247939 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:21.247835 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c5e03e2-611d-4ef6-804d-a769a4148613-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68d996c557t8mkr\" (UID: \"0c5e03e2-611d-4ef6-804d-a769a4148613\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557t8mkr" Oct 31 01:04:21.247939 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:21.247913 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c5e03e2-611d-4ef6-804d-a769a4148613-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68d996c557t8mkr\" (UID: \"0c5e03e2-611d-4ef6-804d-a769a4148613\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557t8mkr" Oct 31 01:04:21.248496 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:21.248442 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c5e03e2-611d-4ef6-804d-a769a4148613-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68d996c557t8mkr\" (UID: \"0c5e03e2-611d-4ef6-804d-a769a4148613\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557t8mkr" Oct 31 01:04:21.248603 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:21.248546 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c5e03e2-611d-4ef6-804d-a769a4148613-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68d996c557t8mkr\" (UID: \"0c5e03e2-611d-4ef6-804d-a769a4148613\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557t8mkr" Oct 31 01:04:21.250077 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:21.250052 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c5e03e2-611d-4ef6-804d-a769a4148613-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68d996c557t8mkr\" (UID: \"0c5e03e2-611d-4ef6-804d-a769a4148613\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557t8mkr" Oct 31 01:04:21.250176 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:21.250120 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c5e03e2-611d-4ef6-804d-a769a4148613-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68d996c557t8mkr\" (UID: \"0c5e03e2-611d-4ef6-804d-a769a4148613\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557t8mkr" Oct 31 01:04:21.266741 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:21.266716 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c5e03e2-611d-4ef6-804d-a769a4148613-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68d996c557t8mkr\" (UID: \"0c5e03e2-611d-4ef6-804d-a769a4148613\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557t8mkr" Oct 31 01:04:21.266863 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:21.266836 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsntg\" (UniqueName: \"kubernetes.io/projected/0c5e03e2-611d-4ef6-804d-a769a4148613-kube-api-access-jsntg\") pod \"data-science-gateway-data-science-gateway-class-68d996c557t8mkr\" (UID: \"0c5e03e2-611d-4ef6-804d-a769a4148613\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557t8mkr" Oct 31 01:04:21.304830 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:21.304800 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:21.338601 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:21.338578 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557t8mkr" Oct 31 01:04:21.499120 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:21.499038 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:21.633067 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:21.632979 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557t8mkr"] Oct 31 01:04:21.635800 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:04:21.635773 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c5e03e2_611d_4ef6_804d_a769a4148613.slice/crio-2bdbd4b021b86ed3e8c56922dcc30c821262865fafd60668a3f2192b9a7324cf WatchSource:0}: Error finding container 2bdbd4b021b86ed3e8c56922dcc30c821262865fafd60668a3f2192b9a7324cf: Status 404 returned error can't find the container with id 2bdbd4b021b86ed3e8c56922dcc30c821262865fafd60668a3f2192b9a7324cf Oct 31 01:04:21.638348 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:21.638312 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:04:21.638693 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:21.638396 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:04:21.638693 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:21.638468 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:04:22.425202 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:22.425164 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557t8mkr" event={"ID":"0c5e03e2-611d-4ef6-804d-a769a4148613","Type":"ContainerStarted","Data":"fa584e1c3aed5d38f7bd491d156a63645380f558fe86f575be8e2201e6e48f9f"} Oct 31 01:04:22.425402 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:22.425224 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557t8mkr" event={"ID":"0c5e03e2-611d-4ef6-804d-a769a4148613","Type":"ContainerStarted","Data":"2bdbd4b021b86ed3e8c56922dcc30c821262865fafd60668a3f2192b9a7324cf"} Oct 31 01:04:22.432963 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:22.432923 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:22.448787 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:22.448726 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557t8mkr" podStartSLOduration=2.448710921 podStartE2EDuration="2.448710921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:22.447640721 +0000 UTC m=+506.806363037" watchObservedRunningTime="2026-10-31 01:04:22.448710921 +0000 UTC m=+506.807433238" Oct 31 01:04:22.934479 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:22.934253 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:23.094327 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:23.094278 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:23.339145 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:23.339060 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557t8mkr" Oct 31 01:04:23.344223 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:23.344196 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557t8mkr" Oct 31 01:04:23.370985 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:23.370958 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-qw7v5" Oct 31 01:04:23.431163 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:23.431124 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557t8mkr" Oct 31 01:04:23.432085 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:23.432062 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557t8mkr" Oct 31 01:04:24.695642 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:24.695604 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-j86rr"] Oct 31 01:04:24.698437 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:24.698388 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j86rr" Oct 31 01:04:24.711977 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:24.711789 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-q2gdc\"" Oct 31 01:04:24.711977 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:24.711886 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 31 01:04:24.711977 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:24.711940 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 31 01:04:24.713399 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:24.713332 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-j86rr"] Oct 31 01:04:24.715565 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:24.715542 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 31 01:04:24.776914 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:24.776871 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d7e25779-b4c1-46ae-954f-55d8f221a802-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-j86rr\" (UID: \"d7e25779-b4c1-46ae-954f-55d8f221a802\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j86rr" Oct 31 01:04:24.777094 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:24.776942 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7e25779-b4c1-46ae-954f-55d8f221a802-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-j86rr\" (UID: \"d7e25779-b4c1-46ae-954f-55d8f221a802\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j86rr" Oct 31 01:04:24.777161 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:24.777143 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whpjg\" (UniqueName: \"kubernetes.io/projected/d7e25779-b4c1-46ae-954f-55d8f221a802-kube-api-access-whpjg\") pod \"kserve-module-controller-manager-7566cf5fc8-j86rr\" (UID: \"d7e25779-b4c1-46ae-954f-55d8f221a802\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j86rr" Oct 31 01:04:24.777295 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:24.777229 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7e25779-b4c1-46ae-954f-55d8f221a802-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-j86rr\" (UID: \"d7e25779-b4c1-46ae-954f-55d8f221a802\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j86rr" Oct 31 01:04:24.785348 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:24.785217 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:24.878650 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:24.878606 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7e25779-b4c1-46ae-954f-55d8f221a802-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-j86rr\" (UID: \"d7e25779-b4c1-46ae-954f-55d8f221a802\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j86rr" Oct 31 01:04:24.878848 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:24.878695 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d7e25779-b4c1-46ae-954f-55d8f221a802-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-j86rr\" (UID: \"d7e25779-b4c1-46ae-954f-55d8f221a802\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j86rr" Oct 31 01:04:24.878848 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:24.878729 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7e25779-b4c1-46ae-954f-55d8f221a802-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-j86rr\" (UID: \"d7e25779-b4c1-46ae-954f-55d8f221a802\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j86rr" Oct 31 01:04:24.878848 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:24.878779 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whpjg\" (UniqueName: \"kubernetes.io/projected/d7e25779-b4c1-46ae-954f-55d8f221a802-kube-api-access-whpjg\") pod \"kserve-module-controller-manager-7566cf5fc8-j86rr\" (UID: \"d7e25779-b4c1-46ae-954f-55d8f221a802\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j86rr" Oct 31 01:04:24.879167 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:24.879107 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7e25779-b4c1-46ae-954f-55d8f221a802-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-j86rr\" (UID: \"d7e25779-b4c1-46ae-954f-55d8f221a802\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j86rr" Oct 31 01:04:24.879293 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:24.879199 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d7e25779-b4c1-46ae-954f-55d8f221a802-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-j86rr\" (UID: \"d7e25779-b4c1-46ae-954f-55d8f221a802\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j86rr" Oct 31 01:04:24.879800 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:24.879776 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7e25779-b4c1-46ae-954f-55d8f221a802-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-j86rr\" (UID: \"d7e25779-b4c1-46ae-954f-55d8f221a802\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j86rr" Oct 31 01:04:24.890584 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:24.890553 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whpjg\" (UniqueName: \"kubernetes.io/projected/d7e25779-b4c1-46ae-954f-55d8f221a802-kube-api-access-whpjg\") pod \"kserve-module-controller-manager-7566cf5fc8-j86rr\" (UID: \"d7e25779-b4c1-46ae-954f-55d8f221a802\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j86rr" Oct 31 01:04:25.015514 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:25.015405 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j86rr" Oct 31 01:04:25.336187 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:25.336162 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-j86rr"] Oct 31 01:04:25.337959 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:04:25.337929 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e25779_b4c1_46ae_954f_55d8f221a802.slice/crio-a64a69cf63fa14425c29774816436452719d04ca9e09bf1535c20b1ceddf7080 WatchSource:0}: Error finding container a64a69cf63fa14425c29774816436452719d04ca9e09bf1535c20b1ceddf7080: Status 404 returned error can't find the container with id a64a69cf63fa14425c29774816436452719d04ca9e09bf1535c20b1ceddf7080 Oct 31 01:04:25.444162 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:25.444123 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5thg5" event={"ID":"9166216f-113a-4d6a-972e-1d0fc5d36ebb","Type":"ContainerStarted","Data":"49ac1270b7493e146d27663fdba994bed8f994ae086f7867d68aa1f2ebf49d95"} Oct 31 01:04:25.444355 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:25.444190 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5thg5" Oct 31 01:04:25.445479 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:25.445451 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j86rr" event={"ID":"d7e25779-b4c1-46ae-954f-55d8f221a802","Type":"ContainerStarted","Data":"a64a69cf63fa14425c29774816436452719d04ca9e09bf1535c20b1ceddf7080"} Oct 31 01:04:25.473225 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:25.473167 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5thg5" podStartSLOduration=1.341545978 podStartE2EDuration="10.473152264s" totalImagesPullingTime="9.131606286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:25.469520213 +0000 UTC m=+509.828242529" watchObservedRunningTime="2026-10-31 01:04:25.473152264 +0000 UTC m=+509.831874577" Oct 31 01:04:26.587606 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:26.587571 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:26.792338 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:26.792303 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:26.933063 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:26.933030 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:27.687476 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:27.687335 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:27.863536 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:27.863470 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:28.084089 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:28.083962 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:28.200883 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:28.200843 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:04:28.205936 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:28.205907 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-6x2mj" Oct 31 01:04:28.220453 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:28.219207 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-22f5d\"" Oct 31 01:04:28.220453 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:28.219356 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 31 01:04:28.222156 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:28.222128 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:04:28.235730 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:28.235694 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:04:28.317124 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:28.317086 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/09059326-b484-4ed1-b50f-a4b0d9dfbb44-config-file\") pod \"limitador-limitador-7c9b87d574-6x2mj\" (UID: \"09059326-b484-4ed1-b50f-a4b0d9dfbb44\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-6x2mj" Oct 31 01:04:28.317316 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:28.317146 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hbl4\" (UniqueName: \"kubernetes.io/projected/09059326-b484-4ed1-b50f-a4b0d9dfbb44-kube-api-access-6hbl4\") pod \"limitador-limitador-7c9b87d574-6x2mj\" (UID: \"09059326-b484-4ed1-b50f-a4b0d9dfbb44\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-6x2mj" Oct 31 01:04:28.325303 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:28.325273 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-rnh95"] Oct 31 01:04:28.330569 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:28.330543 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rnh95" Oct 31 01:04:28.354284 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:28.354258 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 31 01:04:28.354514 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:28.354339 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 31 01:04:28.357239 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:28.357212 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-rnh95"] Oct 31 01:04:28.419207 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:28.419172 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1840b081-35b8-47de-bdd9-5000bed2b2e7-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-rnh95\" (UID: \"1840b081-35b8-47de-bdd9-5000bed2b2e7\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rnh95" Oct 31 01:04:28.419390 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:28.419233 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1840b081-35b8-47de-bdd9-5000bed2b2e7-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-rnh95\" (UID: \"1840b081-35b8-47de-bdd9-5000bed2b2e7\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rnh95" Oct 31 01:04:28.419390 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:28.419329 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/09059326-b484-4ed1-b50f-a4b0d9dfbb44-config-file\") pod \"limitador-limitador-7c9b87d574-6x2mj\" (UID: \"09059326-b484-4ed1-b50f-a4b0d9dfbb44\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-6x2mj" Oct 31 01:04:28.419390 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:28.419372 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hbl4\" (UniqueName: \"kubernetes.io/projected/09059326-b484-4ed1-b50f-a4b0d9dfbb44-kube-api-access-6hbl4\") pod \"limitador-limitador-7c9b87d574-6x2mj\" (UID: \"09059326-b484-4ed1-b50f-a4b0d9dfbb44\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-6x2mj" Oct 31 01:04:28.419581 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:28.419503 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r864\" (UniqueName: \"kubernetes.io/projected/1840b081-35b8-47de-bdd9-5000bed2b2e7-kube-api-access-6r864\") pod \"kuadrant-console-plugin-66c4fb575d-rnh95\" (UID: \"1840b081-35b8-47de-bdd9-5000bed2b2e7\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rnh95" Oct 31 01:04:28.420142 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:28.420117 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/09059326-b484-4ed1-b50f-a4b0d9dfbb44-config-file\") pod \"limitador-limitador-7c9b87d574-6x2mj\" (UID: \"09059326-b484-4ed1-b50f-a4b0d9dfbb44\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-6x2mj" Oct 31 01:04:28.430694 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:28.430669 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hbl4\" (UniqueName: \"kubernetes.io/projected/09059326-b484-4ed1-b50f-a4b0d9dfbb44-kube-api-access-6hbl4\") pod \"limitador-limitador-7c9b87d574-6x2mj\" (UID: \"09059326-b484-4ed1-b50f-a4b0d9dfbb44\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-6x2mj" Oct 31 01:04:28.448799 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:28.448753 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:28.520087 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:28.520045 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r864\" (UniqueName: \"kubernetes.io/projected/1840b081-35b8-47de-bdd9-5000bed2b2e7-kube-api-access-6r864\") pod \"kuadrant-console-plugin-66c4fb575d-rnh95\" (UID: \"1840b081-35b8-47de-bdd9-5000bed2b2e7\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rnh95" Oct 31 01:04:28.520268 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:28.520108 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1840b081-35b8-47de-bdd9-5000bed2b2e7-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-rnh95\" (UID: \"1840b081-35b8-47de-bdd9-5000bed2b2e7\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rnh95" Oct 31 01:04:28.520268 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:28.520154 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1840b081-35b8-47de-bdd9-5000bed2b2e7-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-rnh95\" (UID: \"1840b081-35b8-47de-bdd9-5000bed2b2e7\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rnh95" Oct 31 01:04:28.520949 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:28.520924 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1840b081-35b8-47de-bdd9-5000bed2b2e7-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-rnh95\" (UID: \"1840b081-35b8-47de-bdd9-5000bed2b2e7\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rnh95" Oct 31 01:04:28.522871 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:28.522849 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1840b081-35b8-47de-bdd9-5000bed2b2e7-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-rnh95\" (UID: \"1840b081-35b8-47de-bdd9-5000bed2b2e7\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rnh95" Oct 31 01:04:28.523653 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:28.523631 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-6x2mj" Oct 31 01:04:28.532884 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:28.532863 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r864\" (UniqueName: \"kubernetes.io/projected/1840b081-35b8-47de-bdd9-5000bed2b2e7-kube-api-access-6r864\") pod \"kuadrant-console-plugin-66c4fb575d-rnh95\" (UID: \"1840b081-35b8-47de-bdd9-5000bed2b2e7\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rnh95" Oct 31 01:04:28.633575 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:28.633484 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:28.655190 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:28.655155 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rnh95" Oct 31 01:04:28.878502 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:04:28.878472 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09059326_b484_4ed1_b50f_a4b0d9dfbb44.slice/crio-5a028086f4e639cf09fbeaa02fd65ce9f309935a4e6f67aef550ab62441ac692 WatchSource:0}: Error finding container 5a028086f4e639cf09fbeaa02fd65ce9f309935a4e6f67aef550ab62441ac692: Status 404 returned error can't find the container with id 5a028086f4e639cf09fbeaa02fd65ce9f309935a4e6f67aef550ab62441ac692 Oct 31 01:04:28.884315 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:28.884248 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:04:28.983808 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:28.983779 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-rnh95"] Oct 31 01:04:28.985689 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:04:28.985662 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1840b081_35b8_47de_bdd9_5000bed2b2e7.slice/crio-c7d540c15e30828e5a43b5f0bde36bacc9799ae21deae8429ad13b399bfefe56 WatchSource:0}: Error finding container c7d540c15e30828e5a43b5f0bde36bacc9799ae21deae8429ad13b399bfefe56: Status 404 returned error can't find the container with id c7d540c15e30828e5a43b5f0bde36bacc9799ae21deae8429ad13b399bfefe56 Oct 31 01:04:29.149768 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:29.149689 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:29.326795 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:29.326753 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:29.474350 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:29.474316 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-6x2mj" event={"ID":"09059326-b484-4ed1-b50f-a4b0d9dfbb44","Type":"ContainerStarted","Data":"5a028086f4e639cf09fbeaa02fd65ce9f309935a4e6f67aef550ab62441ac692"} Oct 31 01:04:29.475898 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:29.475861 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j86rr" event={"ID":"d7e25779-b4c1-46ae-954f-55d8f221a802","Type":"ContainerStarted","Data":"91450daaba372611a27ae1bf6ed515a67f84515b3a69cf465fd9bea482e7bee9"} Oct 31 01:04:29.476045 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:29.476013 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j86rr" Oct 31 01:04:29.477088 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:29.477068 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rnh95" event={"ID":"1840b081-35b8-47de-bdd9-5000bed2b2e7","Type":"ContainerStarted","Data":"c7d540c15e30828e5a43b5f0bde36bacc9799ae21deae8429ad13b399bfefe56"} Oct 31 01:04:30.484008 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:30.483917 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-6x2mj" event={"ID":"09059326-b484-4ed1-b50f-a4b0d9dfbb44","Type":"ContainerStarted","Data":"6bc249d6259313b4be02974eb61dce372562939a7144752300472a5d84162d0c"} Oct 31 01:04:30.504182 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:30.504130 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-6x2mj" podStartSLOduration=1.16354812 podStartE2EDuration="2.504115526s" totalImagesPullingTime="1.340567406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:30.502822878 +0000 UTC m=+514.861545196" watchObservedRunningTime="2026-10-31 01:04:30.504115526 +0000 UTC m=+514.862837842" Oct 31 01:04:30.504499 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:30.504472 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j86rr" podStartSLOduration=2.9101812750000002 podStartE2EDuration="6.504461357s" totalImagesPullingTime="3.594280082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:29.509020554 +0000 UTC m=+513.867742870" watchObservedRunningTime="2026-10-31 01:04:30.504461357 +0000 UTC m=+514.863183693" Oct 31 01:04:31.009658 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:04:31.009622 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:04:31.489623 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:31.489595 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-6x2mj" Oct 31 01:04:31.846604 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:31.846504 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:32.012553 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:32.012519 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:32.473581 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:32.473546 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:32.605154 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:32.605120 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:33.077883 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:33.077844 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:33.241518 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:33.241475 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:34.288711 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:34.288674 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:34.441440 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:34.441390 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:35.513859 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:35.513819 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rnh95" event={"ID":"1840b081-35b8-47de-bdd9-5000bed2b2e7","Type":"ContainerStarted","Data":"e2582177e66e941b4f95d12ad80cdbd07d878aa65d832b29a7ffd7f8d5c458ed"} Oct 31 01:04:35.539180 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:35.539113 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rnh95" podStartSLOduration=1.970893835 podStartE2EDuration="7.539099649s" totalImagesPullingTime="5.568205814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:35.538284278 +0000 UTC m=+519.897006594" watchObservedRunningTime="2026-10-31 01:04:35.539099649 +0000 UTC m=+519.897821964" Oct 31 01:04:36.453634 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:36.453605 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5thg5" Oct 31 01:04:36.718004 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:36.717926 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zgxj9"] Oct 31 01:04:36.722953 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:36.722933 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zgxj9" Oct 31 01:04:36.741314 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:36.741287 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zgxj9"] Oct 31 01:04:36.794027 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:36.793995 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c12334b-92c6-4d1c-b5a4-5ad834c842ff-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-zgxj9\" (UID: \"5c12334b-92c6-4d1c-b5a4-5ad834c842ff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zgxj9" Oct 31 01:04:36.794175 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:36.794110 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4424\" (UniqueName: \"kubernetes.io/projected/5c12334b-92c6-4d1c-b5a4-5ad834c842ff-kube-api-access-l4424\") pod \"kuadrant-operator-controller-manager-678d87766f-zgxj9\" (UID: \"5c12334b-92c6-4d1c-b5a4-5ad834c842ff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zgxj9" Oct 31 01:04:36.895614 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:36.895571 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c12334b-92c6-4d1c-b5a4-5ad834c842ff-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-zgxj9\" (UID: \"5c12334b-92c6-4d1c-b5a4-5ad834c842ff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zgxj9" Oct 31 01:04:36.895813 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:36.895706 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4424\" (UniqueName: \"kubernetes.io/projected/5c12334b-92c6-4d1c-b5a4-5ad834c842ff-kube-api-access-l4424\") pod \"kuadrant-operator-controller-manager-678d87766f-zgxj9\" (UID: \"5c12334b-92c6-4d1c-b5a4-5ad834c842ff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zgxj9" Oct 31 01:04:36.896049 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:36.896025 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c12334b-92c6-4d1c-b5a4-5ad834c842ff-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-zgxj9\" (UID: \"5c12334b-92c6-4d1c-b5a4-5ad834c842ff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zgxj9" Oct 31 01:04:36.904553 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:36.904523 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4424\" (UniqueName: \"kubernetes.io/projected/5c12334b-92c6-4d1c-b5a4-5ad834c842ff-kube-api-access-l4424\") pod \"kuadrant-operator-controller-manager-678d87766f-zgxj9\" (UID: \"5c12334b-92c6-4d1c-b5a4-5ad834c842ff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zgxj9" Oct 31 01:04:37.034486 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:37.034371 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zgxj9" Oct 31 01:04:37.316088 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:37.316058 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zgxj9"] Oct 31 01:04:37.318478 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:04:37.318448 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c12334b_92c6_4d1c_b5a4_5ad834c842ff.slice/crio-286eb6702f6e69fbe15ff111228804d53ef8a22b7962f16a4325bc0a223561c8 WatchSource:0}: Error finding container 286eb6702f6e69fbe15ff111228804d53ef8a22b7962f16a4325bc0a223561c8: Status 404 returned error can't find the container with id 286eb6702f6e69fbe15ff111228804d53ef8a22b7962f16a4325bc0a223561c8 Oct 31 01:04:37.526300 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:37.526255 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zgxj9" event={"ID":"5c12334b-92c6-4d1c-b5a4-5ad834c842ff","Type":"ContainerStarted","Data":"5d844e8f5015cb9a4d93d3357384f45ff1fdece9bf65440cd36c8c0d424d5f3d"} Oct 31 01:04:37.526514 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:37.526314 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zgxj9" event={"ID":"5c12334b-92c6-4d1c-b5a4-5ad834c842ff","Type":"ContainerStarted","Data":"286eb6702f6e69fbe15ff111228804d53ef8a22b7962f16a4325bc0a223561c8"} Oct 31 01:04:37.526514 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:37.526399 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zgxj9" Oct 31 01:04:37.549281 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:37.549230 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zgxj9" podStartSLOduration=1.5492163030000001 podStartE2EDuration="1.549216303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:37.547121951 +0000 UTC m=+521.905844303" watchObservedRunningTime="2026-10-31 01:04:37.549216303 +0000 UTC m=+521.907938619" Oct 31 01:04:40.448041 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:40.448002 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:40.554565 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:40.554527 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:40.855704 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:40.855611 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:40.977462 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:40.977408 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:42.496192 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:42.496164 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-6x2mj" Oct 31 01:04:44.048692 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:44.048655 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:44.163446 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:44.163382 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:45.466994 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:45.466961 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:45.623062 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:45.623017 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:47.420802 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:47.420759 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:48.535432 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:48.535388 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zgxj9" Oct 31 01:04:48.572640 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:48.572608 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:48.593595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:48.593566 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5thg5"] Oct 31 01:04:48.593821 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:48.593799 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5thg5" podUID="9166216f-113a-4d6a-972e-1d0fc5d36ebb" containerName="manager" containerID="cri-o://49ac1270b7493e146d27663fdba994bed8f994ae086f7867d68aa1f2ebf49d95" gracePeriod=10 Oct 31 01:04:48.846277 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:48.846246 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5thg5" Oct 31 01:04:49.002138 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:49.002108 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9166216f-113a-4d6a-972e-1d0fc5d36ebb-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9166216f-113a-4d6a-972e-1d0fc5d36ebb-extensions-socket-volume\") pod \"9166216f-113a-4d6a-972e-1d0fc5d36ebb\" (UID: \"9166216f-113a-4d6a-972e-1d0fc5d36ebb\") " Oct 31 01:04:49.002318 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:49.002159 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9166216f-113a-4d6a-972e-1d0fc5d36ebb-kube-api-access-9qnfv\" (UniqueName: \"kubernetes.io/projected/9166216f-113a-4d6a-972e-1d0fc5d36ebb-kube-api-access-9qnfv\") pod \"9166216f-113a-4d6a-972e-1d0fc5d36ebb\" (UID: \"9166216f-113a-4d6a-972e-1d0fc5d36ebb\") " Oct 31 01:04:49.002622 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:49.002600 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9166216f-113a-4d6a-972e-1d0fc5d36ebb-extensions-socket-volume" pod "9166216f-113a-4d6a-972e-1d0fc5d36ebb" (UID: "9166216f-113a-4d6a-972e-1d0fc5d36ebb"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:04:49.004464 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:49.004435 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9166216f-113a-4d6a-972e-1d0fc5d36ebb-kube-api-access-9qnfv" pod "9166216f-113a-4d6a-972e-1d0fc5d36ebb" (UID: "9166216f-113a-4d6a-972e-1d0fc5d36ebb"). InnerVolumeSpecName "kube-api-access-9qnfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:04:49.103244 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:49.103149 4440 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9166216f-113a-4d6a-972e-1d0fc5d36ebb-extensions-socket-volume\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:04:49.103244 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:49.103187 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qnfv\" (UniqueName: \"kubernetes.io/projected/9166216f-113a-4d6a-972e-1d0fc5d36ebb-kube-api-access-9qnfv\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:04:49.283455 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:49.283391 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:49.437734 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:49.437695 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:49.596206 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:49.596171 4440 generic.go:356] "Generic (PLEG): container finished" podID="9166216f-113a-4d6a-972e-1d0fc5d36ebb" containerID="49ac1270b7493e146d27663fdba994bed8f994ae086f7867d68aa1f2ebf49d95" exitCode=0 Oct 31 01:04:49.596688 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:49.596219 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5thg5" Oct 31 01:04:49.596688 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:49.596224 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5thg5" event={"ID":"9166216f-113a-4d6a-972e-1d0fc5d36ebb","Type":"ContainerDied","Data":"49ac1270b7493e146d27663fdba994bed8f994ae086f7867d68aa1f2ebf49d95"} Oct 31 01:04:49.596688 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:49.596264 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5thg5" event={"ID":"9166216f-113a-4d6a-972e-1d0fc5d36ebb","Type":"ContainerDied","Data":"76c9b65db042d37378b7c1de97386e97e662b150b867aeaf6a143dc77e4e7907"} Oct 31 01:04:49.596688 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:49.596288 4440 scope.go:122] "RemoveContainer" containerID="49ac1270b7493e146d27663fdba994bed8f994ae086f7867d68aa1f2ebf49d95" Oct 31 01:04:49.607632 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:49.607610 4440 scope.go:122] "RemoveContainer" containerID="49ac1270b7493e146d27663fdba994bed8f994ae086f7867d68aa1f2ebf49d95" Oct 31 01:04:49.607980 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:04:49.607950 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49ac1270b7493e146d27663fdba994bed8f994ae086f7867d68aa1f2ebf49d95\": container with ID starting with 49ac1270b7493e146d27663fdba994bed8f994ae086f7867d68aa1f2ebf49d95 not found: ID does not exist" containerID="49ac1270b7493e146d27663fdba994bed8f994ae086f7867d68aa1f2ebf49d95" Oct 31 01:04:49.608081 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:49.607987 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49ac1270b7493e146d27663fdba994bed8f994ae086f7867d68aa1f2ebf49d95"} err="failed to get container status \"49ac1270b7493e146d27663fdba994bed8f994ae086f7867d68aa1f2ebf49d95\": rpc error: code = NotFound desc = could not find container \"49ac1270b7493e146d27663fdba994bed8f994ae086f7867d68aa1f2ebf49d95\": container with ID starting with 49ac1270b7493e146d27663fdba994bed8f994ae086f7867d68aa1f2ebf49d95 not found: ID does not exist" Oct 31 01:04:49.632891 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:49.632858 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5thg5"] Oct 31 01:04:49.636041 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:49.636013 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5thg5"] Oct 31 01:04:49.769645 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:49.769563 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9166216f-113a-4d6a-972e-1d0fc5d36ebb" path="/var/lib/kubelet/pods/9166216f-113a-4d6a-972e-1d0fc5d36ebb/volumes" Oct 31 01:04:50.485761 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:50.485731 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j86rr" Oct 31 01:04:52.448247 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:52.448211 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:52.561325 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:04:52.561288 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:00.271582 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:00.271546 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:00.371526 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:00.371490 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:00.880137 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:00.880101 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:00.985606 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:00.985563 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:00.999803 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:05:00.999767 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:05:01.533414 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:01.533370 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:01.651908 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:01.651878 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:02.968096 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:02.968060 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:03.285713 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:03.285620 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:03.391855 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:03.391821 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:08.484139 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:08.484072 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:08.597562 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:08.597521 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:09.734199 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:09.734165 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:09.847219 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:09.847180 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:09.992615 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:09.992524 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:10.106713 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:10.106674 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:12.247725 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:12.247685 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:12.347842 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:12.347799 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:13.771691 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:13.771637 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:13.947483 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:13.947433 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:14.994725 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:14.994684 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:15.386134 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:15.386049 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:20.149724 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:20.149687 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:20.394697 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:20.394636 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:20.617392 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:20.617353 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:20.767651 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:20.767619 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:21.546245 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:21.546210 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:21.676281 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:21.676244 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:21.792112 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:21.792072 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:30.998616 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:05:30.998574 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:05:44.973243 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:44.973133 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:45.085899 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:45.085858 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:45.515527 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:45.515472 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:46.279478 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:46.279437 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:46.366503 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:46.366472 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-7vl46"] Oct 31 01:05:46.366859 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:46.366845 4440 cpu_manager.go:414] "containerMap: removing container" podUID="9166216f-113a-4d6a-972e-1d0fc5d36ebb" containerName="manager" Oct 31 01:05:46.366906 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:46.366860 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9166216f-113a-4d6a-972e-1d0fc5d36ebb" containerName="manager" Oct 31 01:05:46.366952 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:46.366923 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="9166216f-113a-4d6a-972e-1d0fc5d36ebb" containerName="manager" Oct 31 01:05:46.368993 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:46.368974 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7vl46" Oct 31 01:05:46.374243 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:46.374218 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:05:46.374358 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:46.374278 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:05:46.374358 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:46.374304 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-n5p2t\"" Oct 31 01:05:46.374358 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:46.374344 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 31 01:05:46.374500 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:46.374396 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 01:05:46.380023 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:46.380003 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-7vl46"] Oct 31 01:05:46.399231 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:46.399195 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:46.492764 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:46.492733 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc3fcde9-cfdc-4517-95e3-d31107c91361-maas-api-tls\") pod \"maas-api-6d669c7fcd-7vl46\" (UID: \"fc3fcde9-cfdc-4517-95e3-d31107c91361\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7vl46" Oct 31 01:05:46.492912 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:46.492801 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc3fcde9-cfdc-4517-95e3-d31107c91361-openshift-service-ca\") pod \"maas-api-6d669c7fcd-7vl46\" (UID: \"fc3fcde9-cfdc-4517-95e3-d31107c91361\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7vl46" Oct 31 01:05:46.492912 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:46.492879 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc3fcde9-cfdc-4517-95e3-d31107c91361-metrics-certs\") pod \"maas-api-6d669c7fcd-7vl46\" (UID: \"fc3fcde9-cfdc-4517-95e3-d31107c91361\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7vl46" Oct 31 01:05:46.492912 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:46.492903 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56jj2\" (UniqueName: \"kubernetes.io/projected/fc3fcde9-cfdc-4517-95e3-d31107c91361-kube-api-access-56jj2\") pod \"maas-api-6d669c7fcd-7vl46\" (UID: \"fc3fcde9-cfdc-4517-95e3-d31107c91361\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7vl46" Oct 31 01:05:46.593883 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:46.593814 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc3fcde9-cfdc-4517-95e3-d31107c91361-openshift-service-ca\") pod \"maas-api-6d669c7fcd-7vl46\" (UID: \"fc3fcde9-cfdc-4517-95e3-d31107c91361\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7vl46" Oct 31 01:05:46.593883 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:46.593860 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc3fcde9-cfdc-4517-95e3-d31107c91361-metrics-certs\") pod \"maas-api-6d669c7fcd-7vl46\" (UID: \"fc3fcde9-cfdc-4517-95e3-d31107c91361\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7vl46" Oct 31 01:05:46.594055 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:46.593890 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56jj2\" (UniqueName: \"kubernetes.io/projected/fc3fcde9-cfdc-4517-95e3-d31107c91361-kube-api-access-56jj2\") pod \"maas-api-6d669c7fcd-7vl46\" (UID: \"fc3fcde9-cfdc-4517-95e3-d31107c91361\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7vl46" Oct 31 01:05:46.594055 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:46.593944 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc3fcde9-cfdc-4517-95e3-d31107c91361-maas-api-tls\") pod \"maas-api-6d669c7fcd-7vl46\" (UID: \"fc3fcde9-cfdc-4517-95e3-d31107c91361\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7vl46" Oct 31 01:05:46.594578 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:46.594549 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc3fcde9-cfdc-4517-95e3-d31107c91361-openshift-service-ca\") pod \"maas-api-6d669c7fcd-7vl46\" (UID: \"fc3fcde9-cfdc-4517-95e3-d31107c91361\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7vl46" Oct 31 01:05:46.596459 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:46.596413 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc3fcde9-cfdc-4517-95e3-d31107c91361-metrics-certs\") pod \"maas-api-6d669c7fcd-7vl46\" (UID: \"fc3fcde9-cfdc-4517-95e3-d31107c91361\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7vl46" Oct 31 01:05:46.596564 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:46.596499 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc3fcde9-cfdc-4517-95e3-d31107c91361-maas-api-tls\") pod \"maas-api-6d669c7fcd-7vl46\" (UID: \"fc3fcde9-cfdc-4517-95e3-d31107c91361\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7vl46" Oct 31 01:05:46.610568 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:46.610542 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56jj2\" (UniqueName: \"kubernetes.io/projected/fc3fcde9-cfdc-4517-95e3-d31107c91361-kube-api-access-56jj2\") pod \"maas-api-6d669c7fcd-7vl46\" (UID: \"fc3fcde9-cfdc-4517-95e3-d31107c91361\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7vl46" Oct 31 01:05:46.682565 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:46.682538 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7vl46" Oct 31 01:05:46.958340 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:46.958310 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-7vl46"] Oct 31 01:05:46.960047 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:05:46.960021 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc3fcde9_cfdc_4517_95e3_d31107c91361.slice/crio-21e2aba9a8b79f6cc59dfef64294147587182f51dcdd1a1e405b9de37b232fda WatchSource:0}: Error finding container 21e2aba9a8b79f6cc59dfef64294147587182f51dcdd1a1e405b9de37b232fda: Status 404 returned error can't find the container with id 21e2aba9a8b79f6cc59dfef64294147587182f51dcdd1a1e405b9de37b232fda Oct 31 01:05:47.220015 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:47.219929 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:05:47.891217 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:47.891177 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7vl46" event={"ID":"fc3fcde9-cfdc-4517-95e3-d31107c91361","Type":"ContainerStarted","Data":"21e2aba9a8b79f6cc59dfef64294147587182f51dcdd1a1e405b9de37b232fda"} Oct 31 01:05:50.933988 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:50.933946 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7vl46" event={"ID":"fc3fcde9-cfdc-4517-95e3-d31107c91361","Type":"ContainerStarted","Data":"9b01940c5ef85f22ed7140379069454636a80cde062f74f444c4d7196c27fbc4"} Oct 31 01:05:50.934568 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:50.934024 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7vl46" Oct 31 01:05:50.951372 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:50.951326 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7vl46" podStartSLOduration=1.886658487 podStartE2EDuration="4.951313562s" totalImagesPullingTime="3.064655075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:05:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:05:50.949729308 +0000 UTC m=+595.308451624" watchObservedRunningTime="2026-10-31 01:05:50.951313562 +0000 UTC m=+595.310035877" Oct 31 01:05:56.943568 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:56.943538 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7vl46" Oct 31 01:05:56.965894 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:56.965862 4440 generic.go:356] "Generic (PLEG): container finished" podID="dfa83ced-0142-4d9a-9706-7edb28fda974" containerID="d0284841f74e813cc44cba636b2a11fcecb5e3a84770afb2bfc7b37cbfa6ed5e" exitCode=0 Oct 31 01:05:56.966037 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:56.965945 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" event={"ID":"dfa83ced-0142-4d9a-9706-7edb28fda974","Type":"ContainerDied","Data":"d0284841f74e813cc44cba636b2a11fcecb5e3a84770afb2bfc7b37cbfa6ed5e"} Oct 31 01:05:56.966360 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:56.966339 4440 scope.go:122] "RemoveContainer" containerID="d0284841f74e813cc44cba636b2a11fcecb5e3a84770afb2bfc7b37cbfa6ed5e" Oct 31 01:05:56.967651 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:56.967622 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:05:56.967713 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:56.967698 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:05:56.967755 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:56.967740 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:05:57.421925 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:57.421892 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:57.524446 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:57.524388 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:57.973300 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:57.973266 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" event={"ID":"dfa83ced-0142-4d9a-9706-7edb28fda974","Type":"ContainerStarted","Data":"9293ac4746399959babee5fd6474f04cfed2c5a339ddc8d1cb6c29df1ca7ecda"} Oct 31 01:05:58.457105 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:58.457073 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" Oct 31 01:05:58.457295 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:58.457242 4440 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6rjmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Oct 31 01:05:58.457371 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:58.457314 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" podUID="dfa83ced-0142-4d9a-9706-7edb28fda974" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Oct 31 01:05:59.457287 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:59.457246 4440 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6rjmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Oct 31 01:05:59.457703 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:05:59.457306 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" podUID="dfa83ced-0142-4d9a-9706-7edb28fda974" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Oct 31 01:06:00.456783 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:06:00.456747 4440 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6rjmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Oct 31 01:06:00.456953 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:06:00.456814 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" podUID="dfa83ced-0142-4d9a-9706-7edb28fda974" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Oct 31 01:06:01.031093 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:06:01.031046 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:06:01.456813 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:06:01.456781 4440 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6rjmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Oct 31 01:06:01.456988 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:06:01.456832 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" podUID="dfa83ced-0142-4d9a-9706-7edb28fda974" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Oct 31 01:06:02.457545 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:06:02.457512 4440 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6rjmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Oct 31 01:06:02.457923 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:06:02.457561 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" podUID="dfa83ced-0142-4d9a-9706-7edb28fda974" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Oct 31 01:06:03.461357 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:06:03.461321 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" Oct 31 01:06:03.461876 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:06:03.461853 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" Oct 31 01:06:03.462480 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:06:03.462463 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" Oct 31 01:06:07.987750 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:06:07.987717 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-7vl46"] Oct 31 01:06:07.988236 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:06:07.987982 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7vl46" podUID="fc3fcde9-cfdc-4517-95e3-d31107c91361" containerName="maas-api" containerID="cri-o://9b01940c5ef85f22ed7140379069454636a80cde062f74f444c4d7196c27fbc4" gracePeriod=30 Oct 31 01:06:08.223681 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:06:08.223653 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7vl46" Oct 31 01:06:08.317369 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:06:08.317285 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc3fcde9-cfdc-4517-95e3-d31107c91361-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc3fcde9-cfdc-4517-95e3-d31107c91361-maas-api-tls\") pod \"fc3fcde9-cfdc-4517-95e3-d31107c91361\" (UID: \"fc3fcde9-cfdc-4517-95e3-d31107c91361\") " Oct 31 01:06:08.317369 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:06:08.317319 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc3fcde9-cfdc-4517-95e3-d31107c91361-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc3fcde9-cfdc-4517-95e3-d31107c91361-metrics-certs\") pod \"fc3fcde9-cfdc-4517-95e3-d31107c91361\" (UID: \"fc3fcde9-cfdc-4517-95e3-d31107c91361\") " Oct 31 01:06:08.317369 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:06:08.317370 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc3fcde9-cfdc-4517-95e3-d31107c91361-kube-api-access-56jj2\" (UniqueName: \"kubernetes.io/projected/fc3fcde9-cfdc-4517-95e3-d31107c91361-kube-api-access-56jj2\") pod \"fc3fcde9-cfdc-4517-95e3-d31107c91361\" (UID: \"fc3fcde9-cfdc-4517-95e3-d31107c91361\") " Oct 31 01:06:08.317668 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:06:08.317390 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc3fcde9-cfdc-4517-95e3-d31107c91361-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc3fcde9-cfdc-4517-95e3-d31107c91361-openshift-service-ca\") pod \"fc3fcde9-cfdc-4517-95e3-d31107c91361\" (UID: \"fc3fcde9-cfdc-4517-95e3-d31107c91361\") " Oct 31 01:06:08.317755 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:06:08.317729 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc3fcde9-cfdc-4517-95e3-d31107c91361-openshift-service-ca" pod "fc3fcde9-cfdc-4517-95e3-d31107c91361" (UID: "fc3fcde9-cfdc-4517-95e3-d31107c91361"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:06:08.319624 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:06:08.319599 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc3fcde9-cfdc-4517-95e3-d31107c91361-metrics-certs" pod "fc3fcde9-cfdc-4517-95e3-d31107c91361" (UID: "fc3fcde9-cfdc-4517-95e3-d31107c91361"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:06:08.319725 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:06:08.319696 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc3fcde9-cfdc-4517-95e3-d31107c91361-maas-api-tls" pod "fc3fcde9-cfdc-4517-95e3-d31107c91361" (UID: "fc3fcde9-cfdc-4517-95e3-d31107c91361"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:06:08.319725 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:06:08.319698 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc3fcde9-cfdc-4517-95e3-d31107c91361-kube-api-access-56jj2" pod "fc3fcde9-cfdc-4517-95e3-d31107c91361" (UID: "fc3fcde9-cfdc-4517-95e3-d31107c91361"). InnerVolumeSpecName "kube-api-access-56jj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:08.418372 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:06:08.418341 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc3fcde9-cfdc-4517-95e3-d31107c91361-maas-api-tls\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:08.418372 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:06:08.418366 4440 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc3fcde9-cfdc-4517-95e3-d31107c91361-metrics-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:08.418557 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:06:08.418378 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56jj2\" (UniqueName: \"kubernetes.io/projected/fc3fcde9-cfdc-4517-95e3-d31107c91361-kube-api-access-56jj2\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:08.418557 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:06:08.418388 4440 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc3fcde9-cfdc-4517-95e3-d31107c91361-openshift-service-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:09.027297 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:06:09.027260 4440 generic.go:356] "Generic (PLEG): container finished" podID="fc3fcde9-cfdc-4517-95e3-d31107c91361" containerID="9b01940c5ef85f22ed7140379069454636a80cde062f74f444c4d7196c27fbc4" exitCode=0 Oct 31 01:06:09.027730 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:06:09.027308 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7vl46" Oct 31 01:06:09.027730 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:06:09.027346 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7vl46" event={"ID":"fc3fcde9-cfdc-4517-95e3-d31107c91361","Type":"ContainerDied","Data":"9b01940c5ef85f22ed7140379069454636a80cde062f74f444c4d7196c27fbc4"} Oct 31 01:06:09.027730 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:06:09.027402 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7vl46" event={"ID":"fc3fcde9-cfdc-4517-95e3-d31107c91361","Type":"ContainerDied","Data":"21e2aba9a8b79f6cc59dfef64294147587182f51dcdd1a1e405b9de37b232fda"} Oct 31 01:06:09.027730 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:06:09.027440 4440 scope.go:122] "RemoveContainer" containerID="9b01940c5ef85f22ed7140379069454636a80cde062f74f444c4d7196c27fbc4" Oct 31 01:06:09.036783 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:06:09.036762 4440 scope.go:122] "RemoveContainer" containerID="9b01940c5ef85f22ed7140379069454636a80cde062f74f444c4d7196c27fbc4" Oct 31 01:06:09.037065 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:06:09.037049 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b01940c5ef85f22ed7140379069454636a80cde062f74f444c4d7196c27fbc4\": container with ID starting with 9b01940c5ef85f22ed7140379069454636a80cde062f74f444c4d7196c27fbc4 not found: ID does not exist" containerID="9b01940c5ef85f22ed7140379069454636a80cde062f74f444c4d7196c27fbc4" Oct 31 01:06:09.037135 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:06:09.037079 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b01940c5ef85f22ed7140379069454636a80cde062f74f444c4d7196c27fbc4"} err="failed to get container status \"9b01940c5ef85f22ed7140379069454636a80cde062f74f444c4d7196c27fbc4\": rpc error: code = NotFound desc = could not find container \"9b01940c5ef85f22ed7140379069454636a80cde062f74f444c4d7196c27fbc4\": container with ID starting with 9b01940c5ef85f22ed7140379069454636a80cde062f74f444c4d7196c27fbc4 not found: ID does not exist" Oct 31 01:06:09.049173 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:06:09.049150 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-7vl46"] Oct 31 01:06:09.056936 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:06:09.056904 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-7vl46"] Oct 31 01:06:09.774447 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:06:09.771102 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc3fcde9-cfdc-4517-95e3-d31107c91361" path="/var/lib/kubelet/pods/fc3fcde9-cfdc-4517-95e3-d31107c91361/volumes" Oct 31 01:06:30.490937 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:06:30.490897 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:30.610744 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:06:30.610715 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:30.998337 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:06:30.998301 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:06:45.564509 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:06:45.564462 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:55.922943 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:06:55.922898 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:56.029989 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:06:56.029955 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:01.000634 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:07:01.000596 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:07:02.615257 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:02.615224 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:15.054955 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:15.054876 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:15.566143 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:15.566105 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:07:16.416940 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:16.416903 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-n5jxl"] Oct 31 01:07:16.417444 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:16.417410 4440 cpu_manager.go:414] "containerMap: removing container" podUID="fc3fcde9-cfdc-4517-95e3-d31107c91361" containerName="maas-api" Oct 31 01:07:16.417522 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:16.417446 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc3fcde9-cfdc-4517-95e3-d31107c91361" containerName="maas-api" Oct 31 01:07:16.417562 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:16.417539 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc3fcde9-cfdc-4517-95e3-d31107c91361" containerName="maas-api" Oct 31 01:07:16.420901 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:16.420880 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-n5jxl" Oct 31 01:07:16.428524 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:16.428498 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8kl5l\"" Oct 31 01:07:16.428655 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:16.428584 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 31 01:07:16.431848 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:16.431825 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-n5jxl"] Oct 31 01:07:16.458313 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:16.458284 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d2a81a4-1c2e-4e26-8a8a-749d837e1ac1-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-n5jxl\" (UID: \"3d2a81a4-1c2e-4e26-8a8a-749d837e1ac1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-n5jxl" Oct 31 01:07:16.458452 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:16.458323 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d2a81a4-1c2e-4e26-8a8a-749d837e1ac1-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-n5jxl\" (UID: \"3d2a81a4-1c2e-4e26-8a8a-749d837e1ac1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-n5jxl" Oct 31 01:07:16.458452 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:16.458342 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjk2k\" (UniqueName: \"kubernetes.io/projected/3d2a81a4-1c2e-4e26-8a8a-749d837e1ac1-kube-api-access-cjk2k\") pod \"echo-simulator-kserve-64b6f8f96c-n5jxl\" (UID: \"3d2a81a4-1c2e-4e26-8a8a-749d837e1ac1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-n5jxl" Oct 31 01:07:16.458537 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:16.458515 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d2a81a4-1c2e-4e26-8a8a-749d837e1ac1-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-n5jxl\" (UID: \"3d2a81a4-1c2e-4e26-8a8a-749d837e1ac1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-n5jxl" Oct 31 01:07:16.458577 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:16.458557 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d2a81a4-1c2e-4e26-8a8a-749d837e1ac1-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-n5jxl\" (UID: \"3d2a81a4-1c2e-4e26-8a8a-749d837e1ac1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-n5jxl" Oct 31 01:07:16.458610 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:16.458576 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d2a81a4-1c2e-4e26-8a8a-749d837e1ac1-home\") pod \"echo-simulator-kserve-64b6f8f96c-n5jxl\" (UID: \"3d2a81a4-1c2e-4e26-8a8a-749d837e1ac1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-n5jxl" Oct 31 01:07:16.464312 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:16.464288 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 31 01:07:16.464312 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:16.464288 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 31 01:07:16.478360 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:16.478334 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:16.559815 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:16.559781 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d2a81a4-1c2e-4e26-8a8a-749d837e1ac1-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-n5jxl\" (UID: \"3d2a81a4-1c2e-4e26-8a8a-749d837e1ac1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-n5jxl" Oct 31 01:07:16.559815 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:16.559821 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d2a81a4-1c2e-4e26-8a8a-749d837e1ac1-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-n5jxl\" (UID: \"3d2a81a4-1c2e-4e26-8a8a-749d837e1ac1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-n5jxl" Oct 31 01:07:16.560058 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:16.559840 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d2a81a4-1c2e-4e26-8a8a-749d837e1ac1-home\") pod \"echo-simulator-kserve-64b6f8f96c-n5jxl\" (UID: \"3d2a81a4-1c2e-4e26-8a8a-749d837e1ac1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-n5jxl" Oct 31 01:07:16.560058 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:16.559884 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d2a81a4-1c2e-4e26-8a8a-749d837e1ac1-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-n5jxl\" (UID: \"3d2a81a4-1c2e-4e26-8a8a-749d837e1ac1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-n5jxl" Oct 31 01:07:16.560058 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:16.559929 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d2a81a4-1c2e-4e26-8a8a-749d837e1ac1-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-n5jxl\" (UID: \"3d2a81a4-1c2e-4e26-8a8a-749d837e1ac1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-n5jxl" Oct 31 01:07:16.560058 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:16.559954 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjk2k\" (UniqueName: \"kubernetes.io/projected/3d2a81a4-1c2e-4e26-8a8a-749d837e1ac1-kube-api-access-cjk2k\") pod \"echo-simulator-kserve-64b6f8f96c-n5jxl\" (UID: \"3d2a81a4-1c2e-4e26-8a8a-749d837e1ac1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-n5jxl" Oct 31 01:07:16.560311 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:16.560282 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d2a81a4-1c2e-4e26-8a8a-749d837e1ac1-home\") pod \"echo-simulator-kserve-64b6f8f96c-n5jxl\" (UID: \"3d2a81a4-1c2e-4e26-8a8a-749d837e1ac1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-n5jxl" Oct 31 01:07:16.560442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:16.560407 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d2a81a4-1c2e-4e26-8a8a-749d837e1ac1-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-n5jxl\" (UID: \"3d2a81a4-1c2e-4e26-8a8a-749d837e1ac1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-n5jxl" Oct 31 01:07:16.560509 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:16.560447 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d2a81a4-1c2e-4e26-8a8a-749d837e1ac1-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-n5jxl\" (UID: \"3d2a81a4-1c2e-4e26-8a8a-749d837e1ac1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-n5jxl" Oct 31 01:07:16.562400 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:16.562378 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d2a81a4-1c2e-4e26-8a8a-749d837e1ac1-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-n5jxl\" (UID: \"3d2a81a4-1c2e-4e26-8a8a-749d837e1ac1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-n5jxl" Oct 31 01:07:16.562525 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:16.562459 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d2a81a4-1c2e-4e26-8a8a-749d837e1ac1-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-n5jxl\" (UID: \"3d2a81a4-1c2e-4e26-8a8a-749d837e1ac1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-n5jxl" Oct 31 01:07:16.568776 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:16.568753 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjk2k\" (UniqueName: \"kubernetes.io/projected/3d2a81a4-1c2e-4e26-8a8a-749d837e1ac1-kube-api-access-cjk2k\") pod \"echo-simulator-kserve-64b6f8f96c-n5jxl\" (UID: \"3d2a81a4-1c2e-4e26-8a8a-749d837e1ac1\") " pod="llm/echo-simulator-kserve-64b6f8f96c-n5jxl" Oct 31 01:07:16.735403 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:16.735368 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-n5jxl" Oct 31 01:07:16.953366 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:16.953326 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:07:17.015779 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:17.015753 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-n5jxl"] Oct 31 01:07:17.017512 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:07:17.017482 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d2a81a4_1c2e_4e26_8a8a_749d837e1ac1.slice/crio-a475d9929d9d28212159701603bfa8d9623b6a92f8226169ce88676162f2e5e3 WatchSource:0}: Error finding container a475d9929d9d28212159701603bfa8d9623b6a92f8226169ce88676162f2e5e3: Status 404 returned error can't find the container with id a475d9929d9d28212159701603bfa8d9623b6a92f8226169ce88676162f2e5e3 Oct 31 01:07:17.238897 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:17.238862 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:17.360231 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:17.360143 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-n5jxl" event={"ID":"3d2a81a4-1c2e-4e26-8a8a-749d837e1ac1","Type":"ContainerStarted","Data":"a475d9929d9d28212159701603bfa8d9623b6a92f8226169ce88676162f2e5e3"} Oct 31 01:07:17.761381 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:17.761346 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:18.193489 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:18.193457 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd"] Oct 31 01:07:18.199505 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:18.198665 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" Oct 31 01:07:18.211951 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:18.211927 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 31 01:07:18.213252 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:18.213195 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd"] Oct 31 01:07:18.273910 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:18.273875 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bde1a17-6626-433d-a82c-3cff6eac4a1e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd\" (UID: \"2bde1a17-6626-433d-a82c-3cff6eac4a1e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" Oct 31 01:07:18.274093 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:18.273918 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bde1a17-6626-433d-a82c-3cff6eac4a1e-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd\" (UID: \"2bde1a17-6626-433d-a82c-3cff6eac4a1e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" Oct 31 01:07:18.274093 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:18.273946 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6spkt\" (UniqueName: \"kubernetes.io/projected/2bde1a17-6626-433d-a82c-3cff6eac4a1e-kube-api-access-6spkt\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd\" (UID: \"2bde1a17-6626-433d-a82c-3cff6eac4a1e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" Oct 31 01:07:18.274093 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:18.274080 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bde1a17-6626-433d-a82c-3cff6eac4a1e-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd\" (UID: \"2bde1a17-6626-433d-a82c-3cff6eac4a1e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" Oct 31 01:07:18.274223 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:18.274172 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bde1a17-6626-433d-a82c-3cff6eac4a1e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd\" (UID: \"2bde1a17-6626-433d-a82c-3cff6eac4a1e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" Oct 31 01:07:18.274223 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:18.274206 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bde1a17-6626-433d-a82c-3cff6eac4a1e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd\" (UID: \"2bde1a17-6626-433d-a82c-3cff6eac4a1e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" Oct 31 01:07:18.281075 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:18.281046 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:18.375144 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:18.375102 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bde1a17-6626-433d-a82c-3cff6eac4a1e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd\" (UID: \"2bde1a17-6626-433d-a82c-3cff6eac4a1e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" Oct 31 01:07:18.375144 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:18.375136 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bde1a17-6626-433d-a82c-3cff6eac4a1e-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd\" (UID: \"2bde1a17-6626-433d-a82c-3cff6eac4a1e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" Oct 31 01:07:18.375144 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:18.375157 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6spkt\" (UniqueName: \"kubernetes.io/projected/2bde1a17-6626-433d-a82c-3cff6eac4a1e-kube-api-access-6spkt\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd\" (UID: \"2bde1a17-6626-433d-a82c-3cff6eac4a1e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" Oct 31 01:07:18.375612 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:18.375188 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bde1a17-6626-433d-a82c-3cff6eac4a1e-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd\" (UID: \"2bde1a17-6626-433d-a82c-3cff6eac4a1e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" Oct 31 01:07:18.375612 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:18.375259 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bde1a17-6626-433d-a82c-3cff6eac4a1e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd\" (UID: \"2bde1a17-6626-433d-a82c-3cff6eac4a1e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" Oct 31 01:07:18.375612 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:18.375284 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bde1a17-6626-433d-a82c-3cff6eac4a1e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd\" (UID: \"2bde1a17-6626-433d-a82c-3cff6eac4a1e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" Oct 31 01:07:18.375612 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:18.375593 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bde1a17-6626-433d-a82c-3cff6eac4a1e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd\" (UID: \"2bde1a17-6626-433d-a82c-3cff6eac4a1e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" Oct 31 01:07:18.375880 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:18.375647 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bde1a17-6626-433d-a82c-3cff6eac4a1e-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd\" (UID: \"2bde1a17-6626-433d-a82c-3cff6eac4a1e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" Oct 31 01:07:18.375880 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:18.375697 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bde1a17-6626-433d-a82c-3cff6eac4a1e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd\" (UID: \"2bde1a17-6626-433d-a82c-3cff6eac4a1e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" Oct 31 01:07:18.378102 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:18.378079 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bde1a17-6626-433d-a82c-3cff6eac4a1e-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd\" (UID: \"2bde1a17-6626-433d-a82c-3cff6eac4a1e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" Oct 31 01:07:18.378346 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:18.378329 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bde1a17-6626-433d-a82c-3cff6eac4a1e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd\" (UID: \"2bde1a17-6626-433d-a82c-3cff6eac4a1e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" Oct 31 01:07:18.392109 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:18.392080 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6spkt\" (UniqueName: \"kubernetes.io/projected/2bde1a17-6626-433d-a82c-3cff6eac4a1e-kube-api-access-6spkt\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd\" (UID: \"2bde1a17-6626-433d-a82c-3cff6eac4a1e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" Oct 31 01:07:18.512295 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:18.512195 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" Oct 31 01:07:18.799063 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:18.798981 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd"] Oct 31 01:07:18.809763 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:07:18.809731 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2bde1a17_6626_433d_a82c_3cff6eac4a1e.slice/crio-13ce6202ed922e74fb1903c05f797cc9a4f29b5b344dbec7d5e7e073e7d613d8 WatchSource:0}: Error finding container 13ce6202ed922e74fb1903c05f797cc9a4f29b5b344dbec7d5e7e073e7d613d8: Status 404 returned error can't find the container with id 13ce6202ed922e74fb1903c05f797cc9a4f29b5b344dbec7d5e7e073e7d613d8 Oct 31 01:07:19.077078 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:19.077052 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:07:19.376820 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:19.376724 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-n5jxl" event={"ID":"3d2a81a4-1c2e-4e26-8a8a-749d837e1ac1","Type":"ContainerStarted","Data":"42ef794c807f8c3e2653945d4603ad7eec8b1df9d2f0650ee969ba84277740da"} Oct 31 01:07:19.378348 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:19.378320 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" event={"ID":"2bde1a17-6626-433d-a82c-3cff6eac4a1e","Type":"ContainerStarted","Data":"13ce6202ed922e74fb1903c05f797cc9a4f29b5b344dbec7d5e7e073e7d613d8"} Oct 31 01:07:19.395618 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:19.395574 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-n5jxl" podStartSLOduration=1.387750514 podStartE2EDuration="3.395562174s" totalImagesPullingTime="2.00781166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:07:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:07:19.394754138 +0000 UTC m=+683.753476454" watchObservedRunningTime="2026-10-31 01:07:19.395562174 +0000 UTC m=+683.754284489" Oct 31 01:07:19.595948 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:19.595628 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:19.736513 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:19.736384 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-n5jxl" Oct 31 01:07:20.385191 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:20.385147 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" event={"ID":"2bde1a17-6626-433d-a82c-3cff6eac4a1e","Type":"ContainerStarted","Data":"e86c963a538fe6614bc958b32cb09796dc9b04baccf32fe3ff6345f9322dd1d7"} Oct 31 01:07:20.406348 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:20.406288 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podStartSLOduration=1.007792859 podStartE2EDuration="2.406270472s" totalImagesPullingTime="1.398477613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:07:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:07:20.403735696 +0000 UTC m=+684.762458015" watchObservedRunningTime="2026-10-31 01:07:20.406270472 +0000 UTC m=+684.764992789" Oct 31 01:07:20.514498 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:20.514256 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" Oct 31 01:07:20.690935 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:20.690904 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-kr4hj"] Oct 31 01:07:20.694238 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:20.694216 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kr4hj" Oct 31 01:07:20.705459 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:20.703735 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 31 01:07:20.709246 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:20.709218 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-kr4hj"] Oct 31 01:07:20.750777 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:20.750749 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:20.818078 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:20.817993 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76e1f133-7f40-4472-a3f3-c1b6944818d1-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kr4hj\" (UID: \"76e1f133-7f40-4472-a3f3-c1b6944818d1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kr4hj" Oct 31 01:07:20.818078 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:20.818043 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76e1f133-7f40-4472-a3f3-c1b6944818d1-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kr4hj\" (UID: \"76e1f133-7f40-4472-a3f3-c1b6944818d1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kr4hj" Oct 31 01:07:20.818282 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:20.818107 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76e1f133-7f40-4472-a3f3-c1b6944818d1-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kr4hj\" (UID: \"76e1f133-7f40-4472-a3f3-c1b6944818d1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kr4hj" Oct 31 01:07:20.818282 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:20.818155 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76e1f133-7f40-4472-a3f3-c1b6944818d1-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kr4hj\" (UID: \"76e1f133-7f40-4472-a3f3-c1b6944818d1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kr4hj" Oct 31 01:07:20.818282 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:20.818190 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76e1f133-7f40-4472-a3f3-c1b6944818d1-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kr4hj\" (UID: \"76e1f133-7f40-4472-a3f3-c1b6944818d1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kr4hj" Oct 31 01:07:20.818282 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:20.818219 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8p8b\" (UniqueName: \"kubernetes.io/projected/76e1f133-7f40-4472-a3f3-c1b6944818d1-kube-api-access-m8p8b\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kr4hj\" (UID: \"76e1f133-7f40-4472-a3f3-c1b6944818d1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kr4hj" Oct 31 01:07:20.919074 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:20.919039 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8p8b\" (UniqueName: \"kubernetes.io/projected/76e1f133-7f40-4472-a3f3-c1b6944818d1-kube-api-access-m8p8b\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kr4hj\" (UID: \"76e1f133-7f40-4472-a3f3-c1b6944818d1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kr4hj" Oct 31 01:07:20.919255 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:20.919091 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76e1f133-7f40-4472-a3f3-c1b6944818d1-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kr4hj\" (UID: \"76e1f133-7f40-4472-a3f3-c1b6944818d1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kr4hj" Oct 31 01:07:20.919255 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:20.919124 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76e1f133-7f40-4472-a3f3-c1b6944818d1-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kr4hj\" (UID: \"76e1f133-7f40-4472-a3f3-c1b6944818d1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kr4hj" Oct 31 01:07:20.919255 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:20.919182 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76e1f133-7f40-4472-a3f3-c1b6944818d1-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kr4hj\" (UID: \"76e1f133-7f40-4472-a3f3-c1b6944818d1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kr4hj" Oct 31 01:07:20.919255 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:20.919230 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76e1f133-7f40-4472-a3f3-c1b6944818d1-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kr4hj\" (UID: \"76e1f133-7f40-4472-a3f3-c1b6944818d1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kr4hj" Oct 31 01:07:20.919516 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:20.919266 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76e1f133-7f40-4472-a3f3-c1b6944818d1-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kr4hj\" (UID: \"76e1f133-7f40-4472-a3f3-c1b6944818d1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kr4hj" Oct 31 01:07:20.919660 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:20.919611 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76e1f133-7f40-4472-a3f3-c1b6944818d1-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kr4hj\" (UID: \"76e1f133-7f40-4472-a3f3-c1b6944818d1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kr4hj" Oct 31 01:07:20.919784 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:20.919658 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76e1f133-7f40-4472-a3f3-c1b6944818d1-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kr4hj\" (UID: \"76e1f133-7f40-4472-a3f3-c1b6944818d1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kr4hj" Oct 31 01:07:20.919784 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:20.919698 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76e1f133-7f40-4472-a3f3-c1b6944818d1-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kr4hj\" (UID: \"76e1f133-7f40-4472-a3f3-c1b6944818d1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kr4hj" Oct 31 01:07:20.921498 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:20.921466 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76e1f133-7f40-4472-a3f3-c1b6944818d1-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kr4hj\" (UID: \"76e1f133-7f40-4472-a3f3-c1b6944818d1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kr4hj" Oct 31 01:07:20.921770 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:20.921751 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76e1f133-7f40-4472-a3f3-c1b6944818d1-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kr4hj\" (UID: \"76e1f133-7f40-4472-a3f3-c1b6944818d1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kr4hj" Oct 31 01:07:20.951912 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:20.951886 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8p8b\" (UniqueName: \"kubernetes.io/projected/76e1f133-7f40-4472-a3f3-c1b6944818d1-kube-api-access-m8p8b\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kr4hj\" (UID: \"76e1f133-7f40-4472-a3f3-c1b6944818d1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kr4hj" Oct 31 01:07:21.012361 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:21.012330 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kr4hj" Oct 31 01:07:21.293482 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:21.293455 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-kr4hj"] Oct 31 01:07:21.295103 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:07:21.295075 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76e1f133_7f40_4472_a3f3_c1b6944818d1.slice/crio-9d1679ea3084f3dca4286a363208e5f0b58cfe9a0c6dd1baf50c4c144a6d3c44 WatchSource:0}: Error finding container 9d1679ea3084f3dca4286a363208e5f0b58cfe9a0c6dd1baf50c4c144a6d3c44: Status 404 returned error can't find the container with id 9d1679ea3084f3dca4286a363208e5f0b58cfe9a0c6dd1baf50c4c144a6d3c44 Oct 31 01:07:21.391221 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:21.391183 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kr4hj" event={"ID":"76e1f133-7f40-4472-a3f3-c1b6944818d1","Type":"ContainerStarted","Data":"9d1679ea3084f3dca4286a363208e5f0b58cfe9a0c6dd1baf50c4c144a6d3c44"} Oct 31 01:07:22.400861 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:22.400741 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kr4hj" event={"ID":"76e1f133-7f40-4472-a3f3-c1b6944818d1","Type":"ContainerStarted","Data":"a0bf2164efb66fab1fee33ed6343dd93a507f66ade37cfe605a240b32174af68"} Oct 31 01:07:22.419277 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:22.419215 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kr4hj" podStartSLOduration=2.078460572 podStartE2EDuration="2.419195671s" totalImagesPullingTime="340.735099ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:07:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:07:22.417743333 +0000 UTC m=+686.776465650" watchObservedRunningTime="2026-10-31 01:07:22.419195671 +0000 UTC m=+686.777917987" Oct 31 01:07:22.662304 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:22.662221 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:07:23.013446 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:23.013381 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kr4hj" Oct 31 01:07:23.076800 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:23.076768 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-tvltr"] Oct 31 01:07:23.080849 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:23.080826 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tvltr" Oct 31 01:07:23.089607 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:23.089544 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 31 01:07:23.094278 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:23.093518 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-tvltr"] Oct 31 01:07:23.141101 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:23.141067 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c14a3342-f1ee-4856-97ff-4c0cc1e25e19-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tvltr\" (UID: \"c14a3342-f1ee-4856-97ff-4c0cc1e25e19\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tvltr" Oct 31 01:07:23.141101 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:23.141102 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c14a3342-f1ee-4856-97ff-4c0cc1e25e19-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tvltr\" (UID: \"c14a3342-f1ee-4856-97ff-4c0cc1e25e19\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tvltr" Oct 31 01:07:23.141331 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:23.141132 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkp6c\" (UniqueName: \"kubernetes.io/projected/c14a3342-f1ee-4856-97ff-4c0cc1e25e19-kube-api-access-nkp6c\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tvltr\" (UID: \"c14a3342-f1ee-4856-97ff-4c0cc1e25e19\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tvltr" Oct 31 01:07:23.141331 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:23.141279 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c14a3342-f1ee-4856-97ff-4c0cc1e25e19-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tvltr\" (UID: \"c14a3342-f1ee-4856-97ff-4c0cc1e25e19\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tvltr" Oct 31 01:07:23.141466 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:23.141386 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c14a3342-f1ee-4856-97ff-4c0cc1e25e19-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tvltr\" (UID: \"c14a3342-f1ee-4856-97ff-4c0cc1e25e19\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tvltr" Oct 31 01:07:23.141466 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:23.141411 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c14a3342-f1ee-4856-97ff-4c0cc1e25e19-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tvltr\" (UID: \"c14a3342-f1ee-4856-97ff-4c0cc1e25e19\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tvltr" Oct 31 01:07:23.157766 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:23.157733 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:23.242096 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:23.242054 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c14a3342-f1ee-4856-97ff-4c0cc1e25e19-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tvltr\" (UID: \"c14a3342-f1ee-4856-97ff-4c0cc1e25e19\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tvltr" Oct 31 01:07:23.242344 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:23.242104 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkp6c\" (UniqueName: \"kubernetes.io/projected/c14a3342-f1ee-4856-97ff-4c0cc1e25e19-kube-api-access-nkp6c\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tvltr\" (UID: \"c14a3342-f1ee-4856-97ff-4c0cc1e25e19\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tvltr" Oct 31 01:07:23.242344 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:23.242149 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c14a3342-f1ee-4856-97ff-4c0cc1e25e19-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tvltr\" (UID: \"c14a3342-f1ee-4856-97ff-4c0cc1e25e19\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tvltr" Oct 31 01:07:23.242344 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:23.242213 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c14a3342-f1ee-4856-97ff-4c0cc1e25e19-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tvltr\" (UID: \"c14a3342-f1ee-4856-97ff-4c0cc1e25e19\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tvltr" Oct 31 01:07:23.242344 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:23.242236 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c14a3342-f1ee-4856-97ff-4c0cc1e25e19-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tvltr\" (UID: \"c14a3342-f1ee-4856-97ff-4c0cc1e25e19\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tvltr" Oct 31 01:07:23.242344 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:23.242305 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c14a3342-f1ee-4856-97ff-4c0cc1e25e19-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tvltr\" (UID: \"c14a3342-f1ee-4856-97ff-4c0cc1e25e19\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tvltr" Oct 31 01:07:23.242955 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:23.242928 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c14a3342-f1ee-4856-97ff-4c0cc1e25e19-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tvltr\" (UID: \"c14a3342-f1ee-4856-97ff-4c0cc1e25e19\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tvltr" Oct 31 01:07:23.243117 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:23.242929 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c14a3342-f1ee-4856-97ff-4c0cc1e25e19-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tvltr\" (UID: \"c14a3342-f1ee-4856-97ff-4c0cc1e25e19\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tvltr" Oct 31 01:07:23.243117 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:23.242990 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c14a3342-f1ee-4856-97ff-4c0cc1e25e19-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tvltr\" (UID: \"c14a3342-f1ee-4856-97ff-4c0cc1e25e19\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tvltr" Oct 31 01:07:23.244632 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:23.244611 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c14a3342-f1ee-4856-97ff-4c0cc1e25e19-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tvltr\" (UID: \"c14a3342-f1ee-4856-97ff-4c0cc1e25e19\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tvltr" Oct 31 01:07:23.245066 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:23.245049 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c14a3342-f1ee-4856-97ff-4c0cc1e25e19-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tvltr\" (UID: \"c14a3342-f1ee-4856-97ff-4c0cc1e25e19\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tvltr" Oct 31 01:07:23.250950 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:23.250930 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkp6c\" (UniqueName: \"kubernetes.io/projected/c14a3342-f1ee-4856-97ff-4c0cc1e25e19-kube-api-access-nkp6c\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-tvltr\" (UID: \"c14a3342-f1ee-4856-97ff-4c0cc1e25e19\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tvltr" Oct 31 01:07:23.403960 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:23.403865 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tvltr" Oct 31 01:07:23.681996 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:23.681966 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-tvltr"] Oct 31 01:07:23.683840 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:07:23.683806 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc14a3342_f1ee_4856_97ff_4c0cc1e25e19.slice/crio-ac10002f77d20f5da179f3c8b9f69c3480a5410334511be84040d58aebf9f235 WatchSource:0}: Error finding container ac10002f77d20f5da179f3c8b9f69c3480a5410334511be84040d58aebf9f235: Status 404 returned error can't find the container with id ac10002f77d20f5da179f3c8b9f69c3480a5410334511be84040d58aebf9f235 Oct 31 01:07:24.416521 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:24.414227 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tvltr" event={"ID":"c14a3342-f1ee-4856-97ff-4c0cc1e25e19","Type":"ContainerStarted","Data":"932fd51c86ecff87f87a7540bc3c0a1dc360e4eaa27f43fcf58e25eaf6e8154e"} Oct 31 01:07:24.416521 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:24.414301 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tvltr" event={"ID":"c14a3342-f1ee-4856-97ff-4c0cc1e25e19","Type":"ContainerStarted","Data":"ac10002f77d20f5da179f3c8b9f69c3480a5410334511be84040d58aebf9f235"} Oct 31 01:07:24.432779 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:24.432716 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tvltr" podStartSLOduration=1.119679581 podStartE2EDuration="1.432697913s" totalImagesPullingTime="313.018332ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:07:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:07:24.430679914 +0000 UTC m=+688.789402229" watchObservedRunningTime="2026-10-31 01:07:24.432697913 +0000 UTC m=+688.791420230" Oct 31 01:07:25.404942 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:25.404906 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tvltr" Oct 31 01:07:26.089313 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:26.089264 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:26.735869 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:26.735831 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-n5jxl" Oct 31 01:07:26.748637 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:26.748611 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-n5jxl" Oct 31 01:07:26.903022 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:26.902991 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:07:27.449801 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:27.449773 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-n5jxl" Oct 31 01:07:28.512653 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:28.512612 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" Oct 31 01:07:28.526687 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:28.526657 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" Oct 31 01:07:29.461222 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:29.461189 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" Oct 31 01:07:30.999138 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:07:30.999103 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:07:31.012460 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:31.012428 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kr4hj" Oct 31 01:07:31.025155 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:31.025129 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kr4hj" Oct 31 01:07:31.471162 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:31.471136 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kr4hj" Oct 31 01:07:33.404596 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:33.404565 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tvltr" Oct 31 01:07:33.417242 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:33.417221 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tvltr" Oct 31 01:07:33.481904 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:33.481881 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tvltr" Oct 31 01:07:36.564163 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:36.564130 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw"] Oct 31 01:07:36.567921 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:36.567904 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" Oct 31 01:07:36.581796 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:36.581772 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw"] Oct 31 01:07:36.658081 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:36.658053 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56fbc0ea-4056-4f36-867b-68601554c6fb-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-8tjdw\" (UID: \"56fbc0ea-4056-4f36-867b-68601554c6fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" Oct 31 01:07:36.658263 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:36.658090 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56fbc0ea-4056-4f36-867b-68601554c6fb-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-8tjdw\" (UID: \"56fbc0ea-4056-4f36-867b-68601554c6fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" Oct 31 01:07:36.658263 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:36.658121 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56fbc0ea-4056-4f36-867b-68601554c6fb-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-8tjdw\" (UID: \"56fbc0ea-4056-4f36-867b-68601554c6fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" Oct 31 01:07:36.658263 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:36.658226 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56fbc0ea-4056-4f36-867b-68601554c6fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-8tjdw\" (UID: \"56fbc0ea-4056-4f36-867b-68601554c6fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" Oct 31 01:07:36.658263 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:36.658251 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8nxf\" (UniqueName: \"kubernetes.io/projected/56fbc0ea-4056-4f36-867b-68601554c6fb-kube-api-access-z8nxf\") pod \"maas-default-gateway-openshift-default-855f9574cf-8tjdw\" (UID: \"56fbc0ea-4056-4f36-867b-68601554c6fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" Oct 31 01:07:36.658455 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:36.658289 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56fbc0ea-4056-4f36-867b-68601554c6fb-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-8tjdw\" (UID: \"56fbc0ea-4056-4f36-867b-68601554c6fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" Oct 31 01:07:36.658455 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:36.658381 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56fbc0ea-4056-4f36-867b-68601554c6fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-8tjdw\" (UID: \"56fbc0ea-4056-4f36-867b-68601554c6fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" Oct 31 01:07:36.658455 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:36.658436 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56fbc0ea-4056-4f36-867b-68601554c6fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-8tjdw\" (UID: \"56fbc0ea-4056-4f36-867b-68601554c6fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" Oct 31 01:07:36.658569 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:36.658459 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56fbc0ea-4056-4f36-867b-68601554c6fb-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-8tjdw\" (UID: \"56fbc0ea-4056-4f36-867b-68601554c6fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" Oct 31 01:07:36.759210 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:36.759177 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56fbc0ea-4056-4f36-867b-68601554c6fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-8tjdw\" (UID: \"56fbc0ea-4056-4f36-867b-68601554c6fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" Oct 31 01:07:36.759388 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:36.759222 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56fbc0ea-4056-4f36-867b-68601554c6fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-8tjdw\" (UID: \"56fbc0ea-4056-4f36-867b-68601554c6fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" Oct 31 01:07:36.759388 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:36.759248 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56fbc0ea-4056-4f36-867b-68601554c6fb-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-8tjdw\" (UID: \"56fbc0ea-4056-4f36-867b-68601554c6fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" Oct 31 01:07:36.759388 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:36.759294 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56fbc0ea-4056-4f36-867b-68601554c6fb-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-8tjdw\" (UID: \"56fbc0ea-4056-4f36-867b-68601554c6fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" Oct 31 01:07:36.759388 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:36.759330 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56fbc0ea-4056-4f36-867b-68601554c6fb-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-8tjdw\" (UID: \"56fbc0ea-4056-4f36-867b-68601554c6fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" Oct 31 01:07:36.759388 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:36.759374 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56fbc0ea-4056-4f36-867b-68601554c6fb-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-8tjdw\" (UID: \"56fbc0ea-4056-4f36-867b-68601554c6fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" Oct 31 01:07:36.759686 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:36.759399 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56fbc0ea-4056-4f36-867b-68601554c6fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-8tjdw\" (UID: \"56fbc0ea-4056-4f36-867b-68601554c6fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" Oct 31 01:07:36.759686 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:36.759438 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8nxf\" (UniqueName: \"kubernetes.io/projected/56fbc0ea-4056-4f36-867b-68601554c6fb-kube-api-access-z8nxf\") pod \"maas-default-gateway-openshift-default-855f9574cf-8tjdw\" (UID: \"56fbc0ea-4056-4f36-867b-68601554c6fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" Oct 31 01:07:36.759686 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:36.759502 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56fbc0ea-4056-4f36-867b-68601554c6fb-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-8tjdw\" (UID: \"56fbc0ea-4056-4f36-867b-68601554c6fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" Oct 31 01:07:36.759843 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:36.759735 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56fbc0ea-4056-4f36-867b-68601554c6fb-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-8tjdw\" (UID: \"56fbc0ea-4056-4f36-867b-68601554c6fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" Oct 31 01:07:36.759843 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:36.759817 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56fbc0ea-4056-4f36-867b-68601554c6fb-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-8tjdw\" (UID: \"56fbc0ea-4056-4f36-867b-68601554c6fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" Oct 31 01:07:36.760015 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:36.759993 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56fbc0ea-4056-4f36-867b-68601554c6fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-8tjdw\" (UID: \"56fbc0ea-4056-4f36-867b-68601554c6fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" Oct 31 01:07:36.760094 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:36.760068 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56fbc0ea-4056-4f36-867b-68601554c6fb-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-8tjdw\" (UID: \"56fbc0ea-4056-4f36-867b-68601554c6fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" Oct 31 01:07:36.760153 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:36.760116 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56fbc0ea-4056-4f36-867b-68601554c6fb-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-8tjdw\" (UID: \"56fbc0ea-4056-4f36-867b-68601554c6fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" Oct 31 01:07:36.761622 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:36.761604 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56fbc0ea-4056-4f36-867b-68601554c6fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-8tjdw\" (UID: \"56fbc0ea-4056-4f36-867b-68601554c6fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" Oct 31 01:07:36.761869 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:36.761851 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56fbc0ea-4056-4f36-867b-68601554c6fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-8tjdw\" (UID: \"56fbc0ea-4056-4f36-867b-68601554c6fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" Oct 31 01:07:36.768646 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:36.768625 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56fbc0ea-4056-4f36-867b-68601554c6fb-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-8tjdw\" (UID: \"56fbc0ea-4056-4f36-867b-68601554c6fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" Oct 31 01:07:36.769071 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:36.769036 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8nxf\" (UniqueName: \"kubernetes.io/projected/56fbc0ea-4056-4f36-867b-68601554c6fb-kube-api-access-z8nxf\") pod \"maas-default-gateway-openshift-default-855f9574cf-8tjdw\" (UID: \"56fbc0ea-4056-4f36-867b-68601554c6fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" Oct 31 01:07:36.882841 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:36.882758 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" Oct 31 01:07:37.162104 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:37.162080 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw"] Oct 31 01:07:37.163915 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:07:37.163879 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56fbc0ea_4056_4f36_867b_68601554c6fb.slice/crio-4d8b176b43a8bbac451e07a102ff1a18738c7b0424df6f5ced5f15652fba48a7 WatchSource:0}: Error finding container 4d8b176b43a8bbac451e07a102ff1a18738c7b0424df6f5ced5f15652fba48a7: Status 404 returned error can't find the container with id 4d8b176b43a8bbac451e07a102ff1a18738c7b0424df6f5ced5f15652fba48a7 Oct 31 01:07:37.166335 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:37.166299 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:07:37.166468 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:37.166360 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:07:37.166468 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:37.166395 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:07:37.494384 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:37.494351 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" event={"ID":"56fbc0ea-4056-4f36-867b-68601554c6fb","Type":"ContainerStarted","Data":"6819c9febcbc38877d5a3b189182cd0e5bef82d26a62e027faf6b668f14438f6"} Oct 31 01:07:37.494553 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:37.494412 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" event={"ID":"56fbc0ea-4056-4f36-867b-68601554c6fb","Type":"ContainerStarted","Data":"4d8b176b43a8bbac451e07a102ff1a18738c7b0424df6f5ced5f15652fba48a7"} Oct 31 01:07:37.516458 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:37.516012 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" podStartSLOduration=1.515992086 podStartE2EDuration="1.515992086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:07:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:07:37.513299952 +0000 UTC m=+701.872022271" watchObservedRunningTime="2026-10-31 01:07:37.515992086 +0000 UTC m=+701.874714402" Oct 31 01:07:37.883133 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:37.883037 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" Oct 31 01:07:38.884755 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:38.884716 4440 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-8tjdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 31 01:07:38.885219 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:38.884790 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" podUID="56fbc0ea-4056-4f36-867b-68601554c6fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 31 01:07:39.883456 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:39.883399 4440 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-8tjdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 31 01:07:39.883634 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:39.883489 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" podUID="56fbc0ea-4056-4f36-867b-68601554c6fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 31 01:07:40.884070 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:40.884030 4440 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-8tjdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 31 01:07:40.884501 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:40.884093 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" podUID="56fbc0ea-4056-4f36-867b-68601554c6fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 31 01:07:41.883811 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:41.883774 4440 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-8tjdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 31 01:07:41.883997 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:41.883829 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" podUID="56fbc0ea-4056-4f36-867b-68601554c6fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 31 01:07:42.883715 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:42.883671 4440 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-8tjdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 31 01:07:42.884226 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:42.883742 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" podUID="56fbc0ea-4056-4f36-867b-68601554c6fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 31 01:07:43.883266 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:43.883230 4440 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-8tjdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 31 01:07:43.883487 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:43.883292 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" podUID="56fbc0ea-4056-4f36-867b-68601554c6fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 31 01:07:45.425940 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:45.425909 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" Oct 31 01:07:45.426453 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:45.426206 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" Oct 31 01:07:45.427287 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:45.427271 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" Oct 31 01:07:45.613131 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:45.613091 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:51.585868 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:51.585829 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc"] Oct 31 01:07:51.589512 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:51.589489 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" Oct 31 01:07:51.603308 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:51.603280 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc"] Oct 31 01:07:51.613099 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:51.613055 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103d9e31-f559-4366-98ab-2c80cb627428-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-qhwdc\" (UID: \"103d9e31-f559-4366-98ab-2c80cb627428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" Oct 31 01:07:51.613254 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:51.613109 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103d9e31-f559-4366-98ab-2c80cb627428-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-qhwdc\" (UID: \"103d9e31-f559-4366-98ab-2c80cb627428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" Oct 31 01:07:51.613254 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:51.613225 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103d9e31-f559-4366-98ab-2c80cb627428-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-qhwdc\" (UID: \"103d9e31-f559-4366-98ab-2c80cb627428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" Oct 31 01:07:51.613363 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:51.613269 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103d9e31-f559-4366-98ab-2c80cb627428-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-qhwdc\" (UID: \"103d9e31-f559-4366-98ab-2c80cb627428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" Oct 31 01:07:51.613363 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:51.613308 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103d9e31-f559-4366-98ab-2c80cb627428-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-qhwdc\" (UID: \"103d9e31-f559-4366-98ab-2c80cb627428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" Oct 31 01:07:51.613507 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:51.613342 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103d9e31-f559-4366-98ab-2c80cb627428-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-qhwdc\" (UID: \"103d9e31-f559-4366-98ab-2c80cb627428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" Oct 31 01:07:51.613551 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:51.613522 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103d9e31-f559-4366-98ab-2c80cb627428-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-qhwdc\" (UID: \"103d9e31-f559-4366-98ab-2c80cb627428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" Oct 31 01:07:51.613626 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:51.613606 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7jn8\" (UniqueName: \"kubernetes.io/projected/103d9e31-f559-4366-98ab-2c80cb627428-kube-api-access-l7jn8\") pod \"maas-default-gateway-openshift-default-855f9574cf-qhwdc\" (UID: \"103d9e31-f559-4366-98ab-2c80cb627428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" Oct 31 01:07:51.613687 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:51.613674 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103d9e31-f559-4366-98ab-2c80cb627428-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-qhwdc\" (UID: \"103d9e31-f559-4366-98ab-2c80cb627428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" Oct 31 01:07:51.714646 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:51.714603 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103d9e31-f559-4366-98ab-2c80cb627428-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-qhwdc\" (UID: \"103d9e31-f559-4366-98ab-2c80cb627428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" Oct 31 01:07:51.714646 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:51.714653 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103d9e31-f559-4366-98ab-2c80cb627428-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-qhwdc\" (UID: \"103d9e31-f559-4366-98ab-2c80cb627428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" Oct 31 01:07:51.714839 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:51.714688 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7jn8\" (UniqueName: \"kubernetes.io/projected/103d9e31-f559-4366-98ab-2c80cb627428-kube-api-access-l7jn8\") pod \"maas-default-gateway-openshift-default-855f9574cf-qhwdc\" (UID: \"103d9e31-f559-4366-98ab-2c80cb627428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" Oct 31 01:07:51.714839 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:51.714720 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103d9e31-f559-4366-98ab-2c80cb627428-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-qhwdc\" (UID: \"103d9e31-f559-4366-98ab-2c80cb627428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" Oct 31 01:07:51.714839 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:51.714770 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103d9e31-f559-4366-98ab-2c80cb627428-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-qhwdc\" (UID: \"103d9e31-f559-4366-98ab-2c80cb627428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" Oct 31 01:07:51.714839 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:51.714797 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103d9e31-f559-4366-98ab-2c80cb627428-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-qhwdc\" (UID: \"103d9e31-f559-4366-98ab-2c80cb627428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" Oct 31 01:07:51.714839 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:51.714826 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103d9e31-f559-4366-98ab-2c80cb627428-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-qhwdc\" (UID: \"103d9e31-f559-4366-98ab-2c80cb627428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" Oct 31 01:07:51.715097 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:51.715031 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103d9e31-f559-4366-98ab-2c80cb627428-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-qhwdc\" (UID: \"103d9e31-f559-4366-98ab-2c80cb627428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" Oct 31 01:07:51.715097 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:51.715082 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103d9e31-f559-4366-98ab-2c80cb627428-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-qhwdc\" (UID: \"103d9e31-f559-4366-98ab-2c80cb627428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" Oct 31 01:07:51.715205 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:51.715150 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103d9e31-f559-4366-98ab-2c80cb627428-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-qhwdc\" (UID: \"103d9e31-f559-4366-98ab-2c80cb627428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" Oct 31 01:07:51.715266 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:51.715206 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103d9e31-f559-4366-98ab-2c80cb627428-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-qhwdc\" (UID: \"103d9e31-f559-4366-98ab-2c80cb627428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" Oct 31 01:07:51.715305 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:51.715288 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103d9e31-f559-4366-98ab-2c80cb627428-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-qhwdc\" (UID: \"103d9e31-f559-4366-98ab-2c80cb627428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" Oct 31 01:07:51.715391 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:51.715371 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103d9e31-f559-4366-98ab-2c80cb627428-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-qhwdc\" (UID: \"103d9e31-f559-4366-98ab-2c80cb627428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" Oct 31 01:07:51.715627 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:51.715600 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103d9e31-f559-4366-98ab-2c80cb627428-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-qhwdc\" (UID: \"103d9e31-f559-4366-98ab-2c80cb627428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" Oct 31 01:07:51.717205 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:51.717183 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103d9e31-f559-4366-98ab-2c80cb627428-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-qhwdc\" (UID: \"103d9e31-f559-4366-98ab-2c80cb627428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" Oct 31 01:07:51.717306 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:51.717292 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103d9e31-f559-4366-98ab-2c80cb627428-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-qhwdc\" (UID: \"103d9e31-f559-4366-98ab-2c80cb627428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" Oct 31 01:07:51.722488 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:51.722463 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103d9e31-f559-4366-98ab-2c80cb627428-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-qhwdc\" (UID: \"103d9e31-f559-4366-98ab-2c80cb627428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" Oct 31 01:07:51.722582 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:51.722505 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7jn8\" (UniqueName: \"kubernetes.io/projected/103d9e31-f559-4366-98ab-2c80cb627428-kube-api-access-l7jn8\") pod \"maas-default-gateway-openshift-default-855f9574cf-qhwdc\" (UID: \"103d9e31-f559-4366-98ab-2c80cb627428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" Oct 31 01:07:51.906043 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:51.905971 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" Oct 31 01:07:52.184888 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:52.184858 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc"] Oct 31 01:07:52.186451 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:07:52.186401 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod103d9e31_f559_4366_98ab_2c80cb627428.slice/crio-f94a895284b294203aac90ed92f9467d2c5b75bde93e268b2cdec983d9c03eb1 WatchSource:0}: Error finding container f94a895284b294203aac90ed92f9467d2c5b75bde93e268b2cdec983d9c03eb1: Status 404 returned error can't find the container with id f94a895284b294203aac90ed92f9467d2c5b75bde93e268b2cdec983d9c03eb1 Oct 31 01:07:52.188689 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:52.188654 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:07:52.188800 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:52.188733 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:07:52.188800 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:52.188778 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:07:52.625721 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:52.625639 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" event={"ID":"103d9e31-f559-4366-98ab-2c80cb627428","Type":"ContainerStarted","Data":"6fdf8e8fed821fa347743453f59f6a9b5062f2359c4bacf218df8c23340bb7d7"} Oct 31 01:07:52.625721 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:52.625697 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" event={"ID":"103d9e31-f559-4366-98ab-2c80cb627428","Type":"ContainerStarted","Data":"f94a895284b294203aac90ed92f9467d2c5b75bde93e268b2cdec983d9c03eb1"} Oct 31 01:07:52.652950 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:52.649319 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" podStartSLOduration=1.6493015180000001 podStartE2EDuration="1.649301518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:07:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:07:52.643738665 +0000 UTC m=+717.002460982" watchObservedRunningTime="2026-10-31 01:07:52.649301518 +0000 UTC m=+717.008023835" Oct 31 01:07:52.915497 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:52.908513 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" Oct 31 01:07:53.920564 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:53.920527 4440 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-qhwdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 31 01:07:53.920971 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:53.920583 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" podUID="103d9e31-f559-4366-98ab-2c80cb627428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 31 01:07:54.913220 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:54.912775 4440 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-qhwdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 31 01:07:54.913220 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:54.912838 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" podUID="103d9e31-f559-4366-98ab-2c80cb627428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 31 01:07:55.920443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:55.916766 4440 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-qhwdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 31 01:07:55.920443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:55.916822 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" podUID="103d9e31-f559-4366-98ab-2c80cb627428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 31 01:07:56.920460 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:56.910760 4440 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-qhwdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 31 01:07:56.920460 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:56.910824 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" podUID="103d9e31-f559-4366-98ab-2c80cb627428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 31 01:07:57.910844 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:57.910802 4440 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-qhwdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 31 01:07:57.911154 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:57.911115 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" podUID="103d9e31-f559-4366-98ab-2c80cb627428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 31 01:07:58.906601 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:58.906564 4440 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-qhwdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 31 01:07:58.907041 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:07:58.906629 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" podUID="103d9e31-f559-4366-98ab-2c80cb627428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 31 01:08:00.359219 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:00.359182 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" Oct 31 01:08:00.359731 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:00.359509 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" Oct 31 01:08:00.360382 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:00.360359 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" Oct 31 01:08:00.999193 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:08:00.999154 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:08:04.089631 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:04.089594 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:08:06.790464 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:06.763285 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:06.801090 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:06.801047 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-sv7sg"] Oct 31 01:08:06.894631 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:06.894594 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-sv7sg"] Oct 31 01:08:06.895103 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:06.894962 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-sv7sg" Oct 31 01:08:06.958013 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:06.957985 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 31 01:08:06.958383 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:06.958365 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-tdpwg\"" Oct 31 01:08:06.958726 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:06.958708 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 31 01:08:06.984480 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:06.982064 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:07.059506 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:07.058439 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d0ccbe9-d311-4159-abf3-a91dd265970c-metrics-certs\") pod \"ai-gateway-controller-5d9cd99f45-sv7sg\" (UID: \"0d0ccbe9-d311-4159-abf3-a91dd265970c\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-sv7sg" Oct 31 01:08:07.059506 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:07.058523 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp66r\" (UniqueName: \"kubernetes.io/projected/0d0ccbe9-d311-4159-abf3-a91dd265970c-kube-api-access-bp66r\") pod \"ai-gateway-controller-5d9cd99f45-sv7sg\" (UID: \"0d0ccbe9-d311-4159-abf3-a91dd265970c\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-sv7sg" Oct 31 01:08:07.159451 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:07.159355 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp66r\" (UniqueName: \"kubernetes.io/projected/0d0ccbe9-d311-4159-abf3-a91dd265970c-kube-api-access-bp66r\") pod \"ai-gateway-controller-5d9cd99f45-sv7sg\" (UID: \"0d0ccbe9-d311-4159-abf3-a91dd265970c\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-sv7sg" Oct 31 01:08:07.159681 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:07.159569 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d0ccbe9-d311-4159-abf3-a91dd265970c-metrics-certs\") pod \"ai-gateway-controller-5d9cd99f45-sv7sg\" (UID: \"0d0ccbe9-d311-4159-abf3-a91dd265970c\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-sv7sg" Oct 31 01:08:07.163495 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:07.163453 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d0ccbe9-d311-4159-abf3-a91dd265970c-metrics-certs\") pod \"ai-gateway-controller-5d9cd99f45-sv7sg\" (UID: \"0d0ccbe9-d311-4159-abf3-a91dd265970c\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-sv7sg" Oct 31 01:08:07.168366 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:07.168328 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp66r\" (UniqueName: \"kubernetes.io/projected/0d0ccbe9-d311-4159-abf3-a91dd265970c-kube-api-access-bp66r\") pod \"ai-gateway-controller-5d9cd99f45-sv7sg\" (UID: \"0d0ccbe9-d311-4159-abf3-a91dd265970c\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-sv7sg" Oct 31 01:08:07.293752 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:07.293708 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-sv7sg" Oct 31 01:08:07.584932 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:07.584903 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-sv7sg"] Oct 31 01:08:07.586832 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:08:07.586799 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d0ccbe9_d311_4159_abf3_a91dd265970c.slice/crio-e818f691f6e9c18323432a19b90730a000f2b94426ff1c8dcd52d7d01251ff7d WatchSource:0}: Error finding container e818f691f6e9c18323432a19b90730a000f2b94426ff1c8dcd52d7d01251ff7d: Status 404 returned error can't find the container with id e818f691f6e9c18323432a19b90730a000f2b94426ff1c8dcd52d7d01251ff7d Oct 31 01:08:07.588190 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:07.588167 4440 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 01:08:07.770076 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:07.770034 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-sv7sg" event={"ID":"0d0ccbe9-d311-4159-abf3-a91dd265970c","Type":"ContainerStarted","Data":"e818f691f6e9c18323432a19b90730a000f2b94426ff1c8dcd52d7d01251ff7d"} Oct 31 01:08:11.793379 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:11.793294 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-sv7sg" event={"ID":"0d0ccbe9-d311-4159-abf3-a91dd265970c","Type":"ContainerStarted","Data":"a2b74949c41a2a8a443cd2f3bbc9cf1136232b90b1282d7220fb44da0a083879"} Oct 31 01:08:11.793777 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:11.793403 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d9cd99f45-sv7sg" Oct 31 01:08:11.811693 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:11.811633 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d9cd99f45-sv7sg" podStartSLOduration=1.938196909 podStartE2EDuration="5.811620178s" totalImagesPullingTime="3.873423269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:11.810259727 +0000 UTC m=+736.168982044" watchObservedRunningTime="2026-10-31 01:08:11.811620178 +0000 UTC m=+736.170342504" Oct 31 01:08:12.275435 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:12.275388 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:12.282441 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:12.282398 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-w2ptj"] Oct 31 01:08:12.286475 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:12.286456 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-w2ptj" Oct 31 01:08:12.300991 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:12.300959 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-w2ptj"] Oct 31 01:08:12.301850 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:12.301820 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 31 01:08:12.302113 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:12.302083 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-9rgpq\"" Oct 31 01:08:12.305349 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:12.305322 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb6z9\" (UniqueName: \"kubernetes.io/projected/e945ee20-e9d8-45e5-9bd3-1fada7c3ca2e-kube-api-access-tb6z9\") pod \"payload-processing-55fbbbf9dc-w2ptj\" (UID: \"e945ee20-e9d8-45e5-9bd3-1fada7c3ca2e\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-w2ptj" Oct 31 01:08:12.305502 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:12.305364 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e945ee20-e9d8-45e5-9bd3-1fada7c3ca2e-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-w2ptj\" (UID: \"e945ee20-e9d8-45e5-9bd3-1fada7c3ca2e\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-w2ptj" Oct 31 01:08:12.321369 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:12.321334 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-dd7nw"] Oct 31 01:08:12.325078 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:12.325054 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dd7nw" Oct 31 01:08:12.337925 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:12.337898 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-dd7nw"] Oct 31 01:08:12.390637 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:12.390597 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:12.419864 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:12.413962 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb6z9\" (UniqueName: \"kubernetes.io/projected/e945ee20-e9d8-45e5-9bd3-1fada7c3ca2e-kube-api-access-tb6z9\") pod \"payload-processing-55fbbbf9dc-w2ptj\" (UID: \"e945ee20-e9d8-45e5-9bd3-1fada7c3ca2e\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-w2ptj" Oct 31 01:08:12.419864 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:12.414032 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e945ee20-e9d8-45e5-9bd3-1fada7c3ca2e-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-w2ptj\" (UID: \"e945ee20-e9d8-45e5-9bd3-1fada7c3ca2e\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-w2ptj" Oct 31 01:08:12.419864 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:12.414082 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3e485e2-8c87-4e5d-bd9f-639702171908-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-dd7nw\" (UID: \"c3e485e2-8c87-4e5d-bd9f-639702171908\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-dd7nw" Oct 31 01:08:12.419864 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:12.414163 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8rqg\" (UniqueName: \"kubernetes.io/projected/c3e485e2-8c87-4e5d-bd9f-639702171908-kube-api-access-n8rqg\") pod \"payload-pre-processing-7b98bb678c-dd7nw\" (UID: \"c3e485e2-8c87-4e5d-bd9f-639702171908\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-dd7nw" Oct 31 01:08:12.419864 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:12.415583 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e945ee20-e9d8-45e5-9bd3-1fada7c3ca2e-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-w2ptj\" (UID: \"e945ee20-e9d8-45e5-9bd3-1fada7c3ca2e\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-w2ptj" Oct 31 01:08:12.424677 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:12.424644 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb6z9\" (UniqueName: \"kubernetes.io/projected/e945ee20-e9d8-45e5-9bd3-1fada7c3ca2e-kube-api-access-tb6z9\") pod \"payload-processing-55fbbbf9dc-w2ptj\" (UID: \"e945ee20-e9d8-45e5-9bd3-1fada7c3ca2e\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-w2ptj" Oct 31 01:08:12.515785 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:12.515752 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3e485e2-8c87-4e5d-bd9f-639702171908-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-dd7nw\" (UID: \"c3e485e2-8c87-4e5d-bd9f-639702171908\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-dd7nw" Oct 31 01:08:12.516105 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:12.515849 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8rqg\" (UniqueName: \"kubernetes.io/projected/c3e485e2-8c87-4e5d-bd9f-639702171908-kube-api-access-n8rqg\") pod \"payload-pre-processing-7b98bb678c-dd7nw\" (UID: \"c3e485e2-8c87-4e5d-bd9f-639702171908\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-dd7nw" Oct 31 01:08:12.516763 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:12.516738 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3e485e2-8c87-4e5d-bd9f-639702171908-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-dd7nw\" (UID: \"c3e485e2-8c87-4e5d-bd9f-639702171908\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-dd7nw" Oct 31 01:08:12.524395 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:12.524370 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8rqg\" (UniqueName: \"kubernetes.io/projected/c3e485e2-8c87-4e5d-bd9f-639702171908-kube-api-access-n8rqg\") pod \"payload-pre-processing-7b98bb678c-dd7nw\" (UID: \"c3e485e2-8c87-4e5d-bd9f-639702171908\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-dd7nw" Oct 31 01:08:12.600757 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:12.600655 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-w2ptj" Oct 31 01:08:12.643655 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:12.643620 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dd7nw" Oct 31 01:08:12.890278 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:12.890249 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-w2ptj"] Oct 31 01:08:12.892204 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:08:12.892173 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode945ee20_e9d8_45e5_9bd3_1fada7c3ca2e.slice/crio-7f74b6921c15ae21a3abd9194c33ec1372f8bf2f8b06d50ca737717deffa981a WatchSource:0}: Error finding container 7f74b6921c15ae21a3abd9194c33ec1372f8bf2f8b06d50ca737717deffa981a: Status 404 returned error can't find the container with id 7f74b6921c15ae21a3abd9194c33ec1372f8bf2f8b06d50ca737717deffa981a Oct 31 01:08:12.935319 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:12.935290 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-dd7nw"] Oct 31 01:08:12.936468 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:08:12.936390 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e485e2_8c87_4e5d_bd9f_639702171908.slice/crio-75dd640bf55a9d4492c249332ce377dac6fcf376a2a8b9637b4f4b155a0d7355 WatchSource:0}: Error finding container 75dd640bf55a9d4492c249332ce377dac6fcf376a2a8b9637b4f4b155a0d7355: Status 404 returned error can't find the container with id 75dd640bf55a9d4492c249332ce377dac6fcf376a2a8b9637b4f4b155a0d7355 Oct 31 01:08:13.808966 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:13.808904 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-w2ptj" event={"ID":"e945ee20-e9d8-45e5-9bd3-1fada7c3ca2e","Type":"ContainerStarted","Data":"7f74b6921c15ae21a3abd9194c33ec1372f8bf2f8b06d50ca737717deffa981a"} Oct 31 01:08:13.810698 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:13.810667 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dd7nw" event={"ID":"c3e485e2-8c87-4e5d-bd9f-639702171908","Type":"ContainerStarted","Data":"75dd640bf55a9d4492c249332ce377dac6fcf376a2a8b9637b4f4b155a0d7355"} Oct 31 01:08:15.827665 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:15.827627 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-w2ptj" event={"ID":"e945ee20-e9d8-45e5-9bd3-1fada7c3ca2e","Type":"ContainerStarted","Data":"1d36b7916faf8e7b995aae5df15674a989b9434ff57889da1a13c9ca99f7be2c"} Oct 31 01:08:15.828077 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:15.827784 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-w2ptj" Oct 31 01:08:15.846586 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:15.846533 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-w2ptj" podStartSLOduration=1.399278051 podStartE2EDuration="3.846517082s" totalImagesPullingTime="2.447239031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:15.843610818 +0000 UTC m=+740.202333148" watchObservedRunningTime="2026-10-31 01:08:15.846517082 +0000 UTC m=+740.205239397" Oct 31 01:08:16.835552 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:16.835517 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dd7nw" event={"ID":"c3e485e2-8c87-4e5d-bd9f-639702171908","Type":"ContainerStarted","Data":"68274ec822c524abe157634ec702af91839989f961f7ea490813975aa3c735e9"} Oct 31 01:08:16.836008 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:16.835846 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dd7nw" Oct 31 01:08:16.853621 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:16.853564 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dd7nw" podStartSLOduration=1.9782547259999999 podStartE2EDuration="4.853549354s" totalImagesPullingTime="2.875294628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:16.851238878 +0000 UTC m=+741.209961194" watchObservedRunningTime="2026-10-31 01:08:16.853549354 +0000 UTC m=+741.212271670" Oct 31 01:08:21.612452 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:21.612405 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j"] Oct 31 01:08:21.616535 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:21.616515 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" Oct 31 01:08:21.626140 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:21.626116 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j"] Oct 31 01:08:21.721890 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:21.721850 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57c830db-af34-4fb2-b933-f85f04afb3c3-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-wk87j\" (UID: \"57c830db-af34-4fb2-b933-f85f04afb3c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" Oct 31 01:08:21.721890 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:21.721893 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57c830db-af34-4fb2-b933-f85f04afb3c3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-wk87j\" (UID: \"57c830db-af34-4fb2-b933-f85f04afb3c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" Oct 31 01:08:21.722116 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:21.721984 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sntpj\" (UniqueName: \"kubernetes.io/projected/57c830db-af34-4fb2-b933-f85f04afb3c3-kube-api-access-sntpj\") pod \"maas-default-gateway-openshift-default-855f9574cf-wk87j\" (UID: \"57c830db-af34-4fb2-b933-f85f04afb3c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" Oct 31 01:08:21.722116 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:21.722019 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c830db-af34-4fb2-b933-f85f04afb3c3-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-wk87j\" (UID: \"57c830db-af34-4fb2-b933-f85f04afb3c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" Oct 31 01:08:21.722116 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:21.722055 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57c830db-af34-4fb2-b933-f85f04afb3c3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-wk87j\" (UID: \"57c830db-af34-4fb2-b933-f85f04afb3c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" Oct 31 01:08:21.722270 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:21.722135 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57c830db-af34-4fb2-b933-f85f04afb3c3-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-wk87j\" (UID: \"57c830db-af34-4fb2-b933-f85f04afb3c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" Oct 31 01:08:21.722270 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:21.722204 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57c830db-af34-4fb2-b933-f85f04afb3c3-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-wk87j\" (UID: \"57c830db-af34-4fb2-b933-f85f04afb3c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" Oct 31 01:08:21.722381 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:21.722306 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57c830db-af34-4fb2-b933-f85f04afb3c3-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-wk87j\" (UID: \"57c830db-af34-4fb2-b933-f85f04afb3c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" Oct 31 01:08:21.722381 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:21.722344 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c830db-af34-4fb2-b933-f85f04afb3c3-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-wk87j\" (UID: \"57c830db-af34-4fb2-b933-f85f04afb3c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" Oct 31 01:08:21.823467 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:21.823392 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57c830db-af34-4fb2-b933-f85f04afb3c3-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-wk87j\" (UID: \"57c830db-af34-4fb2-b933-f85f04afb3c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" Oct 31 01:08:21.823684 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:21.823476 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57c830db-af34-4fb2-b933-f85f04afb3c3-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-wk87j\" (UID: \"57c830db-af34-4fb2-b933-f85f04afb3c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" Oct 31 01:08:21.823684 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:21.823552 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57c830db-af34-4fb2-b933-f85f04afb3c3-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-wk87j\" (UID: \"57c830db-af34-4fb2-b933-f85f04afb3c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" Oct 31 01:08:21.823684 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:21.823585 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c830db-af34-4fb2-b933-f85f04afb3c3-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-wk87j\" (UID: \"57c830db-af34-4fb2-b933-f85f04afb3c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" Oct 31 01:08:21.823684 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:21.823654 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57c830db-af34-4fb2-b933-f85f04afb3c3-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-wk87j\" (UID: \"57c830db-af34-4fb2-b933-f85f04afb3c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" Oct 31 01:08:21.823927 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:21.823711 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57c830db-af34-4fb2-b933-f85f04afb3c3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-wk87j\" (UID: \"57c830db-af34-4fb2-b933-f85f04afb3c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" Oct 31 01:08:21.823927 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:21.823755 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sntpj\" (UniqueName: \"kubernetes.io/projected/57c830db-af34-4fb2-b933-f85f04afb3c3-kube-api-access-sntpj\") pod \"maas-default-gateway-openshift-default-855f9574cf-wk87j\" (UID: \"57c830db-af34-4fb2-b933-f85f04afb3c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" Oct 31 01:08:21.823927 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:21.823785 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c830db-af34-4fb2-b933-f85f04afb3c3-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-wk87j\" (UID: \"57c830db-af34-4fb2-b933-f85f04afb3c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" Oct 31 01:08:21.823927 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:21.823862 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57c830db-af34-4fb2-b933-f85f04afb3c3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-wk87j\" (UID: \"57c830db-af34-4fb2-b933-f85f04afb3c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" Oct 31 01:08:21.824156 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:21.823929 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57c830db-af34-4fb2-b933-f85f04afb3c3-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-wk87j\" (UID: \"57c830db-af34-4fb2-b933-f85f04afb3c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" Oct 31 01:08:21.824156 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:21.823959 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57c830db-af34-4fb2-b933-f85f04afb3c3-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-wk87j\" (UID: \"57c830db-af34-4fb2-b933-f85f04afb3c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" Oct 31 01:08:21.824156 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:21.824007 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c830db-af34-4fb2-b933-f85f04afb3c3-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-wk87j\" (UID: \"57c830db-af34-4fb2-b933-f85f04afb3c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" Oct 31 01:08:21.824328 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:21.824169 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c830db-af34-4fb2-b933-f85f04afb3c3-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-wk87j\" (UID: \"57c830db-af34-4fb2-b933-f85f04afb3c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" Oct 31 01:08:21.824558 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:21.824537 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57c830db-af34-4fb2-b933-f85f04afb3c3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-wk87j\" (UID: \"57c830db-af34-4fb2-b933-f85f04afb3c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" Oct 31 01:08:21.826041 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:21.826012 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57c830db-af34-4fb2-b933-f85f04afb3c3-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-wk87j\" (UID: \"57c830db-af34-4fb2-b933-f85f04afb3c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" Oct 31 01:08:21.826257 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:21.826240 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57c830db-af34-4fb2-b933-f85f04afb3c3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-wk87j\" (UID: \"57c830db-af34-4fb2-b933-f85f04afb3c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" Oct 31 01:08:21.831651 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:21.831619 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57c830db-af34-4fb2-b933-f85f04afb3c3-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-wk87j\" (UID: \"57c830db-af34-4fb2-b933-f85f04afb3c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" Oct 31 01:08:21.831735 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:21.831706 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sntpj\" (UniqueName: \"kubernetes.io/projected/57c830db-af34-4fb2-b933-f85f04afb3c3-kube-api-access-sntpj\") pod \"maas-default-gateway-openshift-default-855f9574cf-wk87j\" (UID: \"57c830db-af34-4fb2-b933-f85f04afb3c3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" Oct 31 01:08:21.930614 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:21.930580 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" Oct 31 01:08:22.214113 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:22.214083 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j"] Oct 31 01:08:22.215512 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:08:22.215483 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57c830db_af34_4fb2_b933_f85f04afb3c3.slice/crio-32c6d1b6b3e35fa617515463d0df513087cbd4e630928a428ae9b817b32d2ec2 WatchSource:0}: Error finding container 32c6d1b6b3e35fa617515463d0df513087cbd4e630928a428ae9b817b32d2ec2: Status 404 returned error can't find the container with id 32c6d1b6b3e35fa617515463d0df513087cbd4e630928a428ae9b817b32d2ec2 Oct 31 01:08:22.217783 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:22.217753 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:08:22.217851 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:22.217817 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:08:22.217851 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:22.217846 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:08:22.816620 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:22.816587 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d9cd99f45-sv7sg" Oct 31 01:08:22.870408 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:22.870373 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" event={"ID":"57c830db-af34-4fb2-b933-f85f04afb3c3","Type":"ContainerStarted","Data":"62a9b4e7d48088b502fa7ead2debf5e226d16818b996d8f844433ca75575a580"} Oct 31 01:08:22.870559 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:22.870453 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" event={"ID":"57c830db-af34-4fb2-b933-f85f04afb3c3","Type":"ContainerStarted","Data":"32c6d1b6b3e35fa617515463d0df513087cbd4e630928a428ae9b817b32d2ec2"} Oct 31 01:08:22.891028 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:22.890975 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" podStartSLOduration=1.890963351 podStartE2EDuration="1.890963351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:22.887625709 +0000 UTC m=+747.246348026" watchObservedRunningTime="2026-10-31 01:08:22.890963351 +0000 UTC m=+747.249685666" Oct 31 01:08:22.931189 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:22.931149 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" Oct 31 01:08:23.069498 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:23.069404 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-569c46f4cf-fzxcn"] Oct 31 01:08:23.073303 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:23.073280 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-569c46f4cf-fzxcn" Oct 31 01:08:23.081569 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:23.081542 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-569c46f4cf-fzxcn"] Oct 31 01:08:23.139223 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:23.139180 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stwht\" (UniqueName: \"kubernetes.io/projected/fa4b150f-8d6f-4ee3-a150-1977819e47d9-kube-api-access-stwht\") pod \"ai-gateway-controller-569c46f4cf-fzxcn\" (UID: \"fa4b150f-8d6f-4ee3-a150-1977819e47d9\") " pod="opendatahub/ai-gateway-controller-569c46f4cf-fzxcn" Oct 31 01:08:23.139452 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:23.139353 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa4b150f-8d6f-4ee3-a150-1977819e47d9-metrics-certs\") pod \"ai-gateway-controller-569c46f4cf-fzxcn\" (UID: \"fa4b150f-8d6f-4ee3-a150-1977819e47d9\") " pod="opendatahub/ai-gateway-controller-569c46f4cf-fzxcn" Oct 31 01:08:23.240221 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:23.240181 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa4b150f-8d6f-4ee3-a150-1977819e47d9-metrics-certs\") pod \"ai-gateway-controller-569c46f4cf-fzxcn\" (UID: \"fa4b150f-8d6f-4ee3-a150-1977819e47d9\") " pod="opendatahub/ai-gateway-controller-569c46f4cf-fzxcn" Oct 31 01:08:23.240412 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:23.240267 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stwht\" (UniqueName: \"kubernetes.io/projected/fa4b150f-8d6f-4ee3-a150-1977819e47d9-kube-api-access-stwht\") pod \"ai-gateway-controller-569c46f4cf-fzxcn\" (UID: \"fa4b150f-8d6f-4ee3-a150-1977819e47d9\") " pod="opendatahub/ai-gateway-controller-569c46f4cf-fzxcn" Oct 31 01:08:23.242740 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:23.242714 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa4b150f-8d6f-4ee3-a150-1977819e47d9-metrics-certs\") pod \"ai-gateway-controller-569c46f4cf-fzxcn\" (UID: \"fa4b150f-8d6f-4ee3-a150-1977819e47d9\") " pod="opendatahub/ai-gateway-controller-569c46f4cf-fzxcn" Oct 31 01:08:23.248928 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:23.248887 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stwht\" (UniqueName: \"kubernetes.io/projected/fa4b150f-8d6f-4ee3-a150-1977819e47d9-kube-api-access-stwht\") pod \"ai-gateway-controller-569c46f4cf-fzxcn\" (UID: \"fa4b150f-8d6f-4ee3-a150-1977819e47d9\") " pod="opendatahub/ai-gateway-controller-569c46f4cf-fzxcn" Oct 31 01:08:23.388070 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:23.387970 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-569c46f4cf-fzxcn" Oct 31 01:08:23.694230 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:23.694199 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-569c46f4cf-fzxcn"] Oct 31 01:08:23.702465 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:08:23.695470 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa4b150f_8d6f_4ee3_a150_1977819e47d9.slice/crio-36a55b6252b21c4aece4f71a6d165fb2581455e7ab41f02c46dd3219aa7878d4 WatchSource:0}: Error finding container 36a55b6252b21c4aece4f71a6d165fb2581455e7ab41f02c46dd3219aa7878d4: Status 404 returned error can't find the container with id 36a55b6252b21c4aece4f71a6d165fb2581455e7ab41f02c46dd3219aa7878d4 Oct 31 01:08:23.877762 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:23.877723 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-569c46f4cf-fzxcn" event={"ID":"fa4b150f-8d6f-4ee3-a150-1977819e47d9","Type":"ContainerStarted","Data":"36a55b6252b21c4aece4f71a6d165fb2581455e7ab41f02c46dd3219aa7878d4"} Oct 31 01:08:23.933025 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:23.932988 4440 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-wk87j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 31 01:08:23.933189 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:23.933053 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" podUID="57c830db-af34-4fb2-b933-f85f04afb3c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 31 01:08:24.884705 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:24.884667 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-569c46f4cf-fzxcn" event={"ID":"fa4b150f-8d6f-4ee3-a150-1977819e47d9","Type":"ContainerStarted","Data":"28cf87b7fad5c3b70996c4ab88ee04bafab5c4709bbd4d1cb00872f30ceddcd9"} Oct 31 01:08:24.885079 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:24.884968 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-569c46f4cf-fzxcn" Oct 31 01:08:24.920684 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:24.920609 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-569c46f4cf-fzxcn" podStartSLOduration=1.459878722 podStartE2EDuration="1.920582726s" totalImagesPullingTime="460.704004ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:24.902865172 +0000 UTC m=+749.261587489" watchObservedRunningTime="2026-10-31 01:08:24.920582726 +0000 UTC m=+749.279305047" Oct 31 01:08:24.932029 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:24.931994 4440 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-wk87j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 31 01:08:24.932189 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:24.932051 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" podUID="57c830db-af34-4fb2-b933-f85f04afb3c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 31 01:08:25.931560 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:25.931520 4440 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-wk87j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 31 01:08:25.931981 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:25.931581 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" podUID="57c830db-af34-4fb2-b933-f85f04afb3c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 31 01:08:26.838839 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:26.838810 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-w2ptj" Oct 31 01:08:26.931974 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:26.931922 4440 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-wk87j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 31 01:08:26.932446 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:26.932005 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" podUID="57c830db-af34-4fb2-b933-f85f04afb3c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 31 01:08:27.844344 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:27.844310 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dd7nw" Oct 31 01:08:27.931988 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:27.931952 4440 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-wk87j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 31 01:08:27.932385 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:27.932018 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" podUID="57c830db-af34-4fb2-b933-f85f04afb3c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 31 01:08:28.930940 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:28.930902 4440 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-wk87j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 31 01:08:28.931265 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:28.930961 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" podUID="57c830db-af34-4fb2-b933-f85f04afb3c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 31 01:08:30.460167 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:30.460132 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" Oct 31 01:08:30.461396 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:30.461364 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" Oct 31 01:08:30.461570 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:30.461454 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" Oct 31 01:08:30.999608 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:08:30.999574 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:08:35.893463 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:35.893432 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-569c46f4cf-fzxcn" Oct 31 01:08:35.942277 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:35.942242 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-sv7sg"] Oct 31 01:08:35.942534 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:35.942509 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5d9cd99f45-sv7sg" podUID="0d0ccbe9-d311-4159-abf3-a91dd265970c" containerName="manager" containerID="cri-o://a2b74949c41a2a8a443cd2f3bbc9cf1136232b90b1282d7220fb44da0a083879" gracePeriod=30 Oct 31 01:08:36.196068 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:36.196041 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-sv7sg" Oct 31 01:08:36.286407 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:36.286372 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d0ccbe9-d311-4159-abf3-a91dd265970c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d0ccbe9-d311-4159-abf3-a91dd265970c-metrics-certs\") pod \"0d0ccbe9-d311-4159-abf3-a91dd265970c\" (UID: \"0d0ccbe9-d311-4159-abf3-a91dd265970c\") " Oct 31 01:08:36.286609 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:36.286566 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d0ccbe9-d311-4159-abf3-a91dd265970c-kube-api-access-bp66r\" (UniqueName: \"kubernetes.io/projected/0d0ccbe9-d311-4159-abf3-a91dd265970c-kube-api-access-bp66r\") pod \"0d0ccbe9-d311-4159-abf3-a91dd265970c\" (UID: \"0d0ccbe9-d311-4159-abf3-a91dd265970c\") " Oct 31 01:08:36.288624 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:36.288586 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d0ccbe9-d311-4159-abf3-a91dd265970c-kube-api-access-bp66r" pod "0d0ccbe9-d311-4159-abf3-a91dd265970c" (UID: "0d0ccbe9-d311-4159-abf3-a91dd265970c"). InnerVolumeSpecName "kube-api-access-bp66r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:08:36.288740 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:36.288636 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d0ccbe9-d311-4159-abf3-a91dd265970c-metrics-certs" pod "0d0ccbe9-d311-4159-abf3-a91dd265970c" (UID: "0d0ccbe9-d311-4159-abf3-a91dd265970c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:08:36.387958 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:36.387927 4440 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d0ccbe9-d311-4159-abf3-a91dd265970c-metrics-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:36.387958 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:36.387957 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp66r\" (UniqueName: \"kubernetes.io/projected/0d0ccbe9-d311-4159-abf3-a91dd265970c-kube-api-access-bp66r\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:36.978919 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:36.978838 4440 generic.go:356] "Generic (PLEG): container finished" podID="0d0ccbe9-d311-4159-abf3-a91dd265970c" containerID="a2b74949c41a2a8a443cd2f3bbc9cf1136232b90b1282d7220fb44da0a083879" exitCode=0 Oct 31 01:08:36.978919 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:36.978888 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-sv7sg" Oct 31 01:08:36.979402 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:36.978921 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-sv7sg" event={"ID":"0d0ccbe9-d311-4159-abf3-a91dd265970c","Type":"ContainerDied","Data":"a2b74949c41a2a8a443cd2f3bbc9cf1136232b90b1282d7220fb44da0a083879"} Oct 31 01:08:36.979402 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:36.978985 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-sv7sg" event={"ID":"0d0ccbe9-d311-4159-abf3-a91dd265970c","Type":"ContainerDied","Data":"e818f691f6e9c18323432a19b90730a000f2b94426ff1c8dcd52d7d01251ff7d"} Oct 31 01:08:36.979402 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:36.979022 4440 scope.go:122] "RemoveContainer" containerID="a2b74949c41a2a8a443cd2f3bbc9cf1136232b90b1282d7220fb44da0a083879" Oct 31 01:08:36.989441 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:36.989404 4440 scope.go:122] "RemoveContainer" containerID="a2b74949c41a2a8a443cd2f3bbc9cf1136232b90b1282d7220fb44da0a083879" Oct 31 01:08:36.989764 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:08:36.989736 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2b74949c41a2a8a443cd2f3bbc9cf1136232b90b1282d7220fb44da0a083879\": container with ID starting with a2b74949c41a2a8a443cd2f3bbc9cf1136232b90b1282d7220fb44da0a083879 not found: ID does not exist" containerID="a2b74949c41a2a8a443cd2f3bbc9cf1136232b90b1282d7220fb44da0a083879" Oct 31 01:08:36.989838 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:36.989777 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2b74949c41a2a8a443cd2f3bbc9cf1136232b90b1282d7220fb44da0a083879"} err="failed to get container status \"a2b74949c41a2a8a443cd2f3bbc9cf1136232b90b1282d7220fb44da0a083879\": rpc error: code = NotFound desc = could not find container \"a2b74949c41a2a8a443cd2f3bbc9cf1136232b90b1282d7220fb44da0a083879\": container with ID starting with a2b74949c41a2a8a443cd2f3bbc9cf1136232b90b1282d7220fb44da0a083879 not found: ID does not exist" Oct 31 01:08:37.000591 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:37.000560 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-sv7sg"] Oct 31 01:08:37.006110 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:37.006079 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-sv7sg"] Oct 31 01:08:37.682316 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:37.682285 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:37.769757 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:37.769721 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d0ccbe9-d311-4159-abf3-a91dd265970c" path="/var/lib/kubelet/pods/0d0ccbe9-d311-4159-abf3-a91dd265970c/volumes" Oct 31 01:08:38.200317 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:38.200279 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:39.842617 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:39.842564 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:39.976169 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:39.976132 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:41.375087 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:41.375046 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:41.484552 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:41.484492 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:45.676780 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:08:45.676744 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:01.005788 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:09:01.005745 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:09:05.790641 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:05.790604 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:05.899757 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:05.899722 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:06.325238 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.325203 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d"] Oct 31 01:09:06.325845 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.325822 4440 cpu_manager.go:414] "containerMap: removing container" podUID="0d0ccbe9-d311-4159-abf3-a91dd265970c" containerName="manager" Oct 31 01:09:06.326199 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.326181 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d0ccbe9-d311-4159-abf3-a91dd265970c" containerName="manager" Oct 31 01:09:06.326485 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.326465 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d0ccbe9-d311-4159-abf3-a91dd265970c" containerName="manager" Oct 31 01:09:06.335004 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.334983 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" Oct 31 01:09:06.339725 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.339706 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-1b8d05-openshift-default-dockercfg-q8z4w\"" Oct 31 01:09:06.348880 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.348849 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d"] Oct 31 01:09:06.363249 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.363208 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a46413-c078-447a-9bbb-d9ea9431469e-istio-podinfo\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d\" (UID: \"81a46413-c078-447a-9bbb-d9ea9431469e\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" Oct 31 01:09:06.363365 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.363259 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a46413-c078-447a-9bbb-d9ea9431469e-istio-envoy\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d\" (UID: \"81a46413-c078-447a-9bbb-d9ea9431469e\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" Oct 31 01:09:06.363365 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.363286 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrg6l\" (UniqueName: \"kubernetes.io/projected/81a46413-c078-447a-9bbb-d9ea9431469e-kube-api-access-hrg6l\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d\" (UID: \"81a46413-c078-447a-9bbb-d9ea9431469e\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" Oct 31 01:09:06.363365 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.363329 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a46413-c078-447a-9bbb-d9ea9431469e-workload-socket\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d\" (UID: \"81a46413-c078-447a-9bbb-d9ea9431469e\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" Oct 31 01:09:06.363559 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.363412 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a46413-c078-447a-9bbb-d9ea9431469e-workload-certs\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d\" (UID: \"81a46413-c078-447a-9bbb-d9ea9431469e\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" Oct 31 01:09:06.363559 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.363484 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a46413-c078-447a-9bbb-d9ea9431469e-istio-data\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d\" (UID: \"81a46413-c078-447a-9bbb-d9ea9431469e\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" Oct 31 01:09:06.363559 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.363511 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81a46413-c078-447a-9bbb-d9ea9431469e-istio-token\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d\" (UID: \"81a46413-c078-447a-9bbb-d9ea9431469e\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" Oct 31 01:09:06.363559 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.363549 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a46413-c078-447a-9bbb-d9ea9431469e-credential-socket\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d\" (UID: \"81a46413-c078-447a-9bbb-d9ea9431469e\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" Oct 31 01:09:06.363773 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.363602 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a46413-c078-447a-9bbb-d9ea9431469e-istiod-ca-cert\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d\" (UID: \"81a46413-c078-447a-9bbb-d9ea9431469e\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" Oct 31 01:09:06.436509 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.436473 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:06.465085 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.465048 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a46413-c078-447a-9bbb-d9ea9431469e-workload-socket\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d\" (UID: \"81a46413-c078-447a-9bbb-d9ea9431469e\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" Oct 31 01:09:06.465257 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.465159 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a46413-c078-447a-9bbb-d9ea9431469e-workload-certs\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d\" (UID: \"81a46413-c078-447a-9bbb-d9ea9431469e\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" Oct 31 01:09:06.465257 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.465214 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a46413-c078-447a-9bbb-d9ea9431469e-istio-data\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d\" (UID: \"81a46413-c078-447a-9bbb-d9ea9431469e\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" Oct 31 01:09:06.465257 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.465240 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81a46413-c078-447a-9bbb-d9ea9431469e-istio-token\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d\" (UID: \"81a46413-c078-447a-9bbb-d9ea9431469e\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" Oct 31 01:09:06.465429 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.465267 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a46413-c078-447a-9bbb-d9ea9431469e-credential-socket\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d\" (UID: \"81a46413-c078-447a-9bbb-d9ea9431469e\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" Oct 31 01:09:06.465429 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.465316 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a46413-c078-447a-9bbb-d9ea9431469e-istiod-ca-cert\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d\" (UID: \"81a46413-c078-447a-9bbb-d9ea9431469e\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" Oct 31 01:09:06.465429 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.465357 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a46413-c078-447a-9bbb-d9ea9431469e-istio-podinfo\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d\" (UID: \"81a46413-c078-447a-9bbb-d9ea9431469e\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" Oct 31 01:09:06.465429 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.465385 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a46413-c078-447a-9bbb-d9ea9431469e-istio-envoy\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d\" (UID: \"81a46413-c078-447a-9bbb-d9ea9431469e\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" Oct 31 01:09:06.465687 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.465437 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrg6l\" (UniqueName: \"kubernetes.io/projected/81a46413-c078-447a-9bbb-d9ea9431469e-kube-api-access-hrg6l\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d\" (UID: \"81a46413-c078-447a-9bbb-d9ea9431469e\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" Oct 31 01:09:06.466240 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.466211 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a46413-c078-447a-9bbb-d9ea9431469e-workload-socket\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d\" (UID: \"81a46413-c078-447a-9bbb-d9ea9431469e\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" Oct 31 01:09:06.466708 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.466682 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a46413-c078-447a-9bbb-d9ea9431469e-workload-certs\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d\" (UID: \"81a46413-c078-447a-9bbb-d9ea9431469e\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" Oct 31 01:09:06.466974 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.466954 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a46413-c078-447a-9bbb-d9ea9431469e-istio-data\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d\" (UID: \"81a46413-c078-447a-9bbb-d9ea9431469e\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" Oct 31 01:09:06.467512 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.467488 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a46413-c078-447a-9bbb-d9ea9431469e-credential-socket\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d\" (UID: \"81a46413-c078-447a-9bbb-d9ea9431469e\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" Oct 31 01:09:06.468227 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.468165 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a46413-c078-447a-9bbb-d9ea9431469e-istiod-ca-cert\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d\" (UID: \"81a46413-c078-447a-9bbb-d9ea9431469e\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" Oct 31 01:09:06.471411 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.471314 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a46413-c078-447a-9bbb-d9ea9431469e-istio-envoy\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d\" (UID: \"81a46413-c078-447a-9bbb-d9ea9431469e\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" Oct 31 01:09:06.480471 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.480398 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a46413-c078-447a-9bbb-d9ea9431469e-istio-podinfo\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d\" (UID: \"81a46413-c078-447a-9bbb-d9ea9431469e\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" Oct 31 01:09:06.481950 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.481925 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrg6l\" (UniqueName: \"kubernetes.io/projected/81a46413-c078-447a-9bbb-d9ea9431469e-kube-api-access-hrg6l\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d\" (UID: \"81a46413-c078-447a-9bbb-d9ea9431469e\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" Oct 31 01:09:06.483410 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.483387 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81a46413-c078-447a-9bbb-d9ea9431469e-istio-token\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d\" (UID: \"81a46413-c078-447a-9bbb-d9ea9431469e\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" Oct 31 01:09:06.521437 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.519590 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz"] Oct 31 01:09:06.524774 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.524750 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" Oct 31 01:09:06.540466 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.540398 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-487a6c-openshift-default-dockercfg-fl6qs\"" Oct 31 01:09:06.542968 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.542946 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz"] Oct 31 01:09:06.566115 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.566090 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-workload-certs\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz\" (UID: \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" Oct 31 01:09:06.566231 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.566127 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-workload-socket\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz\" (UID: \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" Oct 31 01:09:06.566231 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.566150 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvx8c\" (UniqueName: \"kubernetes.io/projected/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-kube-api-access-tvx8c\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz\" (UID: \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" Oct 31 01:09:06.566313 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.566294 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-istio-data\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz\" (UID: \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" Oct 31 01:09:06.566355 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.566326 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-istio-envoy\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz\" (UID: \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" Oct 31 01:09:06.566355 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.566345 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-credential-socket\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz\" (UID: \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" Oct 31 01:09:06.566442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.566379 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-istiod-ca-cert\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz\" (UID: \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" Oct 31 01:09:06.566489 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.566443 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-istio-podinfo\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz\" (UID: \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" Oct 31 01:09:06.566559 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.566545 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-istio-token\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz\" (UID: \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" Oct 31 01:09:06.581388 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.581325 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:06.651653 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.651621 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" Oct 31 01:09:06.667667 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.667638 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-istio-token\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz\" (UID: \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" Oct 31 01:09:06.667771 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.667706 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-workload-certs\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz\" (UID: \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" Oct 31 01:09:06.667771 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.667750 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-workload-socket\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz\" (UID: \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" Oct 31 01:09:06.667878 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.667782 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvx8c\" (UniqueName: \"kubernetes.io/projected/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-kube-api-access-tvx8c\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz\" (UID: \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" Oct 31 01:09:06.667878 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.667832 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-istio-data\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz\" (UID: \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" Oct 31 01:09:06.667878 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.667864 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-istio-envoy\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz\" (UID: \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" Oct 31 01:09:06.668032 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.667892 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-credential-socket\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz\" (UID: \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" Oct 31 01:09:06.668032 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.667947 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-istiod-ca-cert\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz\" (UID: \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" Oct 31 01:09:06.668032 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.667974 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-istio-podinfo\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz\" (UID: \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" Oct 31 01:09:06.669301 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.669268 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-workload-certs\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz\" (UID: \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" Oct 31 01:09:06.669596 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.669571 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-workload-socket\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz\" (UID: \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" Oct 31 01:09:06.670676 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.670097 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-istio-data\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz\" (UID: \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" Oct 31 01:09:06.672073 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.672054 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-istio-podinfo\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz\" (UID: \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" Oct 31 01:09:06.672353 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.672312 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-istiod-ca-cert\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz\" (UID: \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" Oct 31 01:09:06.672688 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.672666 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-credential-socket\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz\" (UID: \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" Oct 31 01:09:06.672931 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.672908 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-istio-envoy\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz\" (UID: \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" Oct 31 01:09:06.677817 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.677789 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvx8c\" (UniqueName: \"kubernetes.io/projected/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-kube-api-access-tvx8c\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz\" (UID: \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" Oct 31 01:09:06.678202 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.678179 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-istio-token\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz\" (UID: \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" Oct 31 01:09:06.720806 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.720742 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:06.736837 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.736802 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl"] Oct 31 01:09:06.752289 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.752047 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" Oct 31 01:09:06.759545 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.758787 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-acdaed-openshift-default-dockercfg-swmgr\"" Oct 31 01:09:06.761308 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.761237 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl"] Oct 31 01:09:06.769095 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.768920 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-istio-data\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl\" (UID: \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" Oct 31 01:09:06.769341 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.769321 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-istio-podinfo\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl\" (UID: \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" Oct 31 01:09:06.769892 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.769402 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-workload-socket\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl\" (UID: \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" Oct 31 01:09:06.769892 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.769515 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-istio-token\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl\" (UID: \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" Oct 31 01:09:06.769892 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.769570 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-workload-certs\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl\" (UID: \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" Oct 31 01:09:06.769892 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.769633 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-istio-envoy\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl\" (UID: \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" Oct 31 01:09:06.769892 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.769700 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-credential-socket\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl\" (UID: \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" Oct 31 01:09:06.769892 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.769729 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qtx2\" (UniqueName: \"kubernetes.io/projected/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-kube-api-access-5qtx2\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl\" (UID: \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" Oct 31 01:09:06.769892 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.769762 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-istiod-ca-cert\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl\" (UID: \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" Oct 31 01:09:06.842867 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.842769 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:06.850618 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.850592 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" Oct 31 01:09:06.870857 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.870831 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-workload-socket\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl\" (UID: \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" Oct 31 01:09:06.871189 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.871054 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-istio-token\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl\" (UID: \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" Oct 31 01:09:06.871189 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.871114 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-workload-certs\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl\" (UID: \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" Oct 31 01:09:06.871322 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.871194 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-istio-envoy\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl\" (UID: \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" Oct 31 01:09:06.871322 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.871265 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-credential-socket\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl\" (UID: \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" Oct 31 01:09:06.871322 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.871294 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qtx2\" (UniqueName: \"kubernetes.io/projected/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-kube-api-access-5qtx2\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl\" (UID: \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" Oct 31 01:09:06.871503 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.871333 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-istiod-ca-cert\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl\" (UID: \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" Oct 31 01:09:06.871503 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.871380 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-istio-data\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl\" (UID: \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" Oct 31 01:09:06.871503 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.871443 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-istio-podinfo\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl\" (UID: \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" Oct 31 01:09:06.872139 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.872033 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-workload-socket\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl\" (UID: \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" Oct 31 01:09:06.872391 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.872364 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-credential-socket\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl\" (UID: \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" Oct 31 01:09:06.872651 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.872629 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-workload-certs\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl\" (UID: \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" Oct 31 01:09:06.872731 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.872715 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-istiod-ca-cert\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl\" (UID: \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" Oct 31 01:09:06.872968 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.872947 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-istio-data\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl\" (UID: \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" Oct 31 01:09:06.874642 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.874623 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-istio-podinfo\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl\" (UID: \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" Oct 31 01:09:06.875160 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.875141 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-istio-envoy\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl\" (UID: \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" Oct 31 01:09:06.889620 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.889578 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-istio-token\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl\" (UID: \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" Oct 31 01:09:06.918035 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.896286 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qtx2\" (UniqueName: \"kubernetes.io/projected/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-kube-api-access-5qtx2\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl\" (UID: \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" Oct 31 01:09:06.970359 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.970328 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:06.992341 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.992303 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d"] Oct 31 01:09:06.997278 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:09:06.997247 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81a46413_c078_447a_9bbb_d9ea9431469e.slice/crio-b736ec712d17344a6e4203dbcfaba1845c5679bb699572c1a52b0f4b6c9036d9 WatchSource:0}: Error finding container b736ec712d17344a6e4203dbcfaba1845c5679bb699572c1a52b0f4b6c9036d9: Status 404 returned error can't find the container with id b736ec712d17344a6e4203dbcfaba1845c5679bb699572c1a52b0f4b6c9036d9 Oct 31 01:09:06.999887 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.999848 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:09:07.000001 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.999934 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:09:07.000001 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:06.999980 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:09:07.073000 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:07.072977 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" Oct 31 01:09:07.080071 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:07.080049 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:07.229399 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:07.229300 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz"] Oct 31 01:09:07.244097 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:07.244047 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:07.247775 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:07.247144 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" event={"ID":"81a46413-c078-447a-9bbb-d9ea9431469e","Type":"ContainerStarted","Data":"cb4b0e121cee63b353d20ad49269a0abf7758c19fdb73b48fd1d8b452a7a720d"} Oct 31 01:09:07.247775 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:07.247222 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" event={"ID":"81a46413-c078-447a-9bbb-d9ea9431469e","Type":"ContainerStarted","Data":"b736ec712d17344a6e4203dbcfaba1845c5679bb699572c1a52b0f4b6c9036d9"} Oct 31 01:09:07.257089 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:07.257052 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:09:07.257181 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:07.257160 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:09:07.257220 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:07.257201 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:09:07.274768 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:07.274724 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" podStartSLOduration=1.274709281 podStartE2EDuration="1.274709281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:07.270583291 +0000 UTC m=+791.629305624" watchObservedRunningTime="2026-10-31 01:09:07.274709281 +0000 UTC m=+791.633431597" Oct 31 01:09:07.360620 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:07.360584 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:07.362739 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:07.362709 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl"] Oct 31 01:09:07.367468 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:09:07.367437 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd3cc4c7_0978_44a4_bec3_cb1d3bd89daf.slice/crio-cf2b60db433d0806c15973332161e6c77219d1272722cc8983ad42d6a5e4c35e WatchSource:0}: Error finding container cf2b60db433d0806c15973332161e6c77219d1272722cc8983ad42d6a5e4c35e: Status 404 returned error can't find the container with id cf2b60db433d0806c15973332161e6c77219d1272722cc8983ad42d6a5e4c35e Oct 31 01:09:07.370143 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:07.370103 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:09:07.370221 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:07.370187 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:09:07.370285 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:07.370254 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:09:07.652529 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:07.652493 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" Oct 31 01:09:08.254520 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:08.254484 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" event={"ID":"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf","Type":"ContainerStarted","Data":"c26e3bab275a94d05499b10f9528f1f51aa858a8dba45c8ba18710f2b1c4811a"} Oct 31 01:09:08.254956 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:08.254539 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" event={"ID":"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf","Type":"ContainerStarted","Data":"cf2b60db433d0806c15973332161e6c77219d1272722cc8983ad42d6a5e4c35e"} Oct 31 01:09:08.256131 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:08.256104 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" event={"ID":"f8555032-1556-4f20-a9bb-7f1dcbf99dcc","Type":"ContainerStarted","Data":"d224ea1ef076b80805213e38b80de1bc55b5b9f72bf1d142e023460bd3573b22"} Oct 31 01:09:08.256278 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:08.256147 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" event={"ID":"f8555032-1556-4f20-a9bb-7f1dcbf99dcc","Type":"ContainerStarted","Data":"a619c6589a03293e2dcd26f57f0707a183d1b8bf009a7199854cab9252a502b4"} Oct 31 01:09:08.276912 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:08.276871 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" podStartSLOduration=2.276859956 podStartE2EDuration="2.276859956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:08.275340457 +0000 UTC m=+792.634062775" watchObservedRunningTime="2026-10-31 01:09:08.276859956 +0000 UTC m=+792.635582272" Oct 31 01:09:08.304836 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:08.304784 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" podStartSLOduration=2.304771132 podStartE2EDuration="2.304771132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:08.303625555 +0000 UTC m=+792.662347871" watchObservedRunningTime="2026-10-31 01:09:08.304771132 +0000 UTC m=+792.663493568" Oct 31 01:09:08.657909 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:08.657829 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" Oct 31 01:09:08.851035 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:08.850997 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" Oct 31 01:09:08.856496 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:08.856469 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" Oct 31 01:09:09.073484 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:09.073452 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" Oct 31 01:09:09.078935 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:09.078909 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" Oct 31 01:09:09.262441 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:09.262378 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" Oct 31 01:09:09.262902 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:09.262451 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" Oct 31 01:09:09.262902 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:09.262478 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" Oct 31 01:09:09.263278 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:09.263255 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" Oct 31 01:09:09.263397 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:09.263312 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" Oct 31 01:09:09.263488 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:09.263468 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" Oct 31 01:09:09.654359 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:09.654326 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:09.815433 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:09.815395 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:10.018725 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:10.018688 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:10.169178 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:10.169140 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:10.301036 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:10.300951 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:11.001319 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.001288 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:11.058782 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.058750 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-1b8d05-5c48d56c8c-swf9d"] Oct 31 01:09:11.062455 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.062413 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1b8d05-5c48d56c8c-swf9d" Oct 31 01:09:11.070039 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.070011 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-1b8d05\"" Oct 31 01:09:11.070191 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.070172 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-1b8d05-dockercfg-j52sf\"" Oct 31 01:09:11.072904 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.072882 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-1b8d05-5c48d56c8c-swf9d"] Oct 31 01:09:11.112114 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.112076 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-1b8d05-6f65d64999-htfv4"] Oct 31 01:09:11.116333 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.116310 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-1b8d05-6f65d64999-htfv4" Oct 31 01:09:11.121980 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.121955 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c3f1628-66ce-4761-80ea-e7654895d913-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-1b8d05-5c48d56c8c-swf9d\" (UID: \"7c3f1628-66ce-4761-80ea-e7654895d913\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1b8d05-5c48d56c8c-swf9d" Oct 31 01:09:11.122152 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.122123 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgzqg\" (UniqueName: \"kubernetes.io/projected/7c3f1628-66ce-4761-80ea-e7654895d913-kube-api-access-cgzqg\") pod \"payload-pre-processing-e2e-worker-w6-1b8d05-5c48d56c8c-swf9d\" (UID: \"7c3f1628-66ce-4761-80ea-e7654895d913\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1b8d05-5c48d56c8c-swf9d" Oct 31 01:09:11.129448 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.129398 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-1b8d05-6f65d64999-htfv4"] Oct 31 01:09:11.150939 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.150906 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:11.222882 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.222843 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k67tj\" (UniqueName: \"kubernetes.io/projected/c80bb0f7-7c51-4344-91d9-a37dc7d21a83-kube-api-access-k67tj\") pod \"payload-processing-e2e-worker-w6-1b8d05-6f65d64999-htfv4\" (UID: \"c80bb0f7-7c51-4344-91d9-a37dc7d21a83\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-1b8d05-6f65d64999-htfv4" Oct 31 01:09:11.223024 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.222994 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c3f1628-66ce-4761-80ea-e7654895d913-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-1b8d05-5c48d56c8c-swf9d\" (UID: \"7c3f1628-66ce-4761-80ea-e7654895d913\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1b8d05-5c48d56c8c-swf9d" Oct 31 01:09:11.223065 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.223042 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80bb0f7-7c51-4344-91d9-a37dc7d21a83-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-1b8d05-6f65d64999-htfv4\" (UID: \"c80bb0f7-7c51-4344-91d9-a37dc7d21a83\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-1b8d05-6f65d64999-htfv4" Oct 31 01:09:11.223232 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.223210 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgzqg\" (UniqueName: \"kubernetes.io/projected/7c3f1628-66ce-4761-80ea-e7654895d913-kube-api-access-cgzqg\") pod \"payload-pre-processing-e2e-worker-w6-1b8d05-5c48d56c8c-swf9d\" (UID: \"7c3f1628-66ce-4761-80ea-e7654895d913\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1b8d05-5c48d56c8c-swf9d" Oct 31 01:09:11.223710 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.223693 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c3f1628-66ce-4761-80ea-e7654895d913-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-1b8d05-5c48d56c8c-swf9d\" (UID: \"7c3f1628-66ce-4761-80ea-e7654895d913\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1b8d05-5c48d56c8c-swf9d" Oct 31 01:09:11.233661 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.233633 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgzqg\" (UniqueName: \"kubernetes.io/projected/7c3f1628-66ce-4761-80ea-e7654895d913-kube-api-access-cgzqg\") pod \"payload-pre-processing-e2e-worker-w6-1b8d05-5c48d56c8c-swf9d\" (UID: \"7c3f1628-66ce-4761-80ea-e7654895d913\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1b8d05-5c48d56c8c-swf9d" Oct 31 01:09:11.279581 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.279489 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:11.324475 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.324435 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80bb0f7-7c51-4344-91d9-a37dc7d21a83-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-1b8d05-6f65d64999-htfv4\" (UID: \"c80bb0f7-7c51-4344-91d9-a37dc7d21a83\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-1b8d05-6f65d64999-htfv4" Oct 31 01:09:11.324966 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.324938 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k67tj\" (UniqueName: \"kubernetes.io/projected/c80bb0f7-7c51-4344-91d9-a37dc7d21a83-kube-api-access-k67tj\") pod \"payload-processing-e2e-worker-w6-1b8d05-6f65d64999-htfv4\" (UID: \"c80bb0f7-7c51-4344-91d9-a37dc7d21a83\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-1b8d05-6f65d64999-htfv4" Oct 31 01:09:11.325297 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.325269 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80bb0f7-7c51-4344-91d9-a37dc7d21a83-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-1b8d05-6f65d64999-htfv4\" (UID: \"c80bb0f7-7c51-4344-91d9-a37dc7d21a83\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-1b8d05-6f65d64999-htfv4" Oct 31 01:09:11.334546 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.334522 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k67tj\" (UniqueName: \"kubernetes.io/projected/c80bb0f7-7c51-4344-91d9-a37dc7d21a83-kube-api-access-k67tj\") pod \"payload-processing-e2e-worker-w6-1b8d05-6f65d64999-htfv4\" (UID: \"c80bb0f7-7c51-4344-91d9-a37dc7d21a83\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-1b8d05-6f65d64999-htfv4" Oct 31 01:09:11.377525 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.377499 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1b8d05-5c48d56c8c-swf9d" Oct 31 01:09:11.432956 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.432914 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-1b8d05-6f65d64999-htfv4" Oct 31 01:09:11.601787 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.601689 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-1b8d05-5fdbf4bd8c-lf5w4"] Oct 31 01:09:11.607010 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.606986 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1b8d05-5fdbf4bd8c-lf5w4" Oct 31 01:09:11.625505 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.623850 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-n5p2t\"" Oct 31 01:09:11.625505 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.624233 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:09:11.625505 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.624297 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-1b8d05-serving-cert\"" Oct 31 01:09:11.625505 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.624507 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:09:11.633708 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.626849 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 01:09:11.633708 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.628737 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-1b8d05-5fdbf4bd8c-lf5w4"] Oct 31 01:09:11.656331 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.656306 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-1b8d05-5c48d56c8c-swf9d"] Oct 31 01:09:11.658893 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:09:11.658866 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c3f1628_66ce_4761_80ea_e7654895d913.slice/crio-4e2d29b911ee825cd3960936aea2135eb9e12328bd9db595f2fd693419c5fa17 WatchSource:0}: Error finding container 4e2d29b911ee825cd3960936aea2135eb9e12328bd9db595f2fd693419c5fa17: Status 404 returned error can't find the container with id 4e2d29b911ee825cd3960936aea2135eb9e12328bd9db595f2fd693419c5fa17 Oct 31 01:09:11.692947 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.692922 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:11.724283 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.724262 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-1b8d05-6f65d64999-htfv4"] Oct 31 01:09:11.725583 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:09:11.725563 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc80bb0f7_7c51_4344_91d9_a37dc7d21a83.slice/crio-a82e12f53fd7789b48fc61cb4fed8384fbd5bb69920514a6be18b7ebcd59917c WatchSource:0}: Error finding container a82e12f53fd7789b48fc61cb4fed8384fbd5bb69920514a6be18b7ebcd59917c: Status 404 returned error can't find the container with id a82e12f53fd7789b48fc61cb4fed8384fbd5bb69920514a6be18b7ebcd59917c Oct 31 01:09:11.730167 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.730144 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/293b71e1-036c-491c-b693-f7acc63bed0f-metrics-certs\") pod \"maas-api-e2e-worker-w6-1b8d05-5fdbf4bd8c-lf5w4\" (UID: \"293b71e1-036c-491c-b693-f7acc63bed0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1b8d05-5fdbf4bd8c-lf5w4" Oct 31 01:09:11.730247 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.730178 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/293b71e1-036c-491c-b693-f7acc63bed0f-maas-api-tls\") pod \"maas-api-e2e-worker-w6-1b8d05-5fdbf4bd8c-lf5w4\" (UID: \"293b71e1-036c-491c-b693-f7acc63bed0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1b8d05-5fdbf4bd8c-lf5w4" Oct 31 01:09:11.730247 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.730211 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4pcb\" (UniqueName: \"kubernetes.io/projected/293b71e1-036c-491c-b693-f7acc63bed0f-kube-api-access-t4pcb\") pod \"maas-api-e2e-worker-w6-1b8d05-5fdbf4bd8c-lf5w4\" (UID: \"293b71e1-036c-491c-b693-f7acc63bed0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1b8d05-5fdbf4bd8c-lf5w4" Oct 31 01:09:11.730536 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.730354 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/293b71e1-036c-491c-b693-f7acc63bed0f-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-1b8d05-5fdbf4bd8c-lf5w4\" (UID: \"293b71e1-036c-491c-b693-f7acc63bed0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1b8d05-5fdbf4bd8c-lf5w4" Oct 31 01:09:11.829401 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.829369 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:11.831539 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.831513 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4pcb\" (UniqueName: \"kubernetes.io/projected/293b71e1-036c-491c-b693-f7acc63bed0f-kube-api-access-t4pcb\") pod \"maas-api-e2e-worker-w6-1b8d05-5fdbf4bd8c-lf5w4\" (UID: \"293b71e1-036c-491c-b693-f7acc63bed0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1b8d05-5fdbf4bd8c-lf5w4" Oct 31 01:09:11.831666 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.831588 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/293b71e1-036c-491c-b693-f7acc63bed0f-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-1b8d05-5fdbf4bd8c-lf5w4\" (UID: \"293b71e1-036c-491c-b693-f7acc63bed0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1b8d05-5fdbf4bd8c-lf5w4" Oct 31 01:09:11.831995 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.831943 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/293b71e1-036c-491c-b693-f7acc63bed0f-metrics-certs\") pod \"maas-api-e2e-worker-w6-1b8d05-5fdbf4bd8c-lf5w4\" (UID: \"293b71e1-036c-491c-b693-f7acc63bed0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1b8d05-5fdbf4bd8c-lf5w4" Oct 31 01:09:11.832087 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.832001 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/293b71e1-036c-491c-b693-f7acc63bed0f-maas-api-tls\") pod \"maas-api-e2e-worker-w6-1b8d05-5fdbf4bd8c-lf5w4\" (UID: \"293b71e1-036c-491c-b693-f7acc63bed0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1b8d05-5fdbf4bd8c-lf5w4" Oct 31 01:09:11.832316 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.832295 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/293b71e1-036c-491c-b693-f7acc63bed0f-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-1b8d05-5fdbf4bd8c-lf5w4\" (UID: \"293b71e1-036c-491c-b693-f7acc63bed0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1b8d05-5fdbf4bd8c-lf5w4" Oct 31 01:09:11.834639 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.834616 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/293b71e1-036c-491c-b693-f7acc63bed0f-maas-api-tls\") pod \"maas-api-e2e-worker-w6-1b8d05-5fdbf4bd8c-lf5w4\" (UID: \"293b71e1-036c-491c-b693-f7acc63bed0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1b8d05-5fdbf4bd8c-lf5w4" Oct 31 01:09:11.835322 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.835301 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/293b71e1-036c-491c-b693-f7acc63bed0f-metrics-certs\") pod \"maas-api-e2e-worker-w6-1b8d05-5fdbf4bd8c-lf5w4\" (UID: \"293b71e1-036c-491c-b693-f7acc63bed0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1b8d05-5fdbf4bd8c-lf5w4" Oct 31 01:09:11.839299 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.839277 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4pcb\" (UniqueName: \"kubernetes.io/projected/293b71e1-036c-491c-b693-f7acc63bed0f-kube-api-access-t4pcb\") pod \"maas-api-e2e-worker-w6-1b8d05-5fdbf4bd8c-lf5w4\" (UID: \"293b71e1-036c-491c-b693-f7acc63bed0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1b8d05-5fdbf4bd8c-lf5w4" Oct 31 01:09:11.926809 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.926782 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-0a6cca-6599d4945f-dlflj"] Oct 31 01:09:11.930626 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.930610 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-0a6cca-6599d4945f-dlflj" Oct 31 01:09:11.932807 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.932782 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1b8d05-5fdbf4bd8c-lf5w4" Oct 31 01:09:11.938604 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.938458 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-0a6cca\"" Oct 31 01:09:11.939313 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.939289 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-0a6cca-6599d4945f-dlflj"] Oct 31 01:09:11.942142 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.942110 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-0a6cca-dockercfg-wxkhl\"" Oct 31 01:09:11.958054 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:11.958026 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:12.034098 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:12.034065 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm6l2\" (UniqueName: \"kubernetes.io/projected/eaad3e7d-1293-43d5-9fe6-674e69ea840e-kube-api-access-nm6l2\") pod \"payload-pre-processing-e2e-worker-w2-0a6cca-6599d4945f-dlflj\" (UID: \"eaad3e7d-1293-43d5-9fe6-674e69ea840e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-0a6cca-6599d4945f-dlflj" Oct 31 01:09:12.034275 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:12.034224 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaad3e7d-1293-43d5-9fe6-674e69ea840e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-0a6cca-6599d4945f-dlflj\" (UID: \"eaad3e7d-1293-43d5-9fe6-674e69ea840e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-0a6cca-6599d4945f-dlflj" Oct 31 01:09:12.136120 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:12.135980 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm6l2\" (UniqueName: \"kubernetes.io/projected/eaad3e7d-1293-43d5-9fe6-674e69ea840e-kube-api-access-nm6l2\") pod \"payload-pre-processing-e2e-worker-w2-0a6cca-6599d4945f-dlflj\" (UID: \"eaad3e7d-1293-43d5-9fe6-674e69ea840e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-0a6cca-6599d4945f-dlflj" Oct 31 01:09:12.136120 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:12.136095 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaad3e7d-1293-43d5-9fe6-674e69ea840e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-0a6cca-6599d4945f-dlflj\" (UID: \"eaad3e7d-1293-43d5-9fe6-674e69ea840e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-0a6cca-6599d4945f-dlflj" Oct 31 01:09:12.137088 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:12.137037 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaad3e7d-1293-43d5-9fe6-674e69ea840e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-0a6cca-6599d4945f-dlflj\" (UID: \"eaad3e7d-1293-43d5-9fe6-674e69ea840e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-0a6cca-6599d4945f-dlflj" Oct 31 01:09:12.163382 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:12.163312 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm6l2\" (UniqueName: \"kubernetes.io/projected/eaad3e7d-1293-43d5-9fe6-674e69ea840e-kube-api-access-nm6l2\") pod \"payload-pre-processing-e2e-worker-w2-0a6cca-6599d4945f-dlflj\" (UID: \"eaad3e7d-1293-43d5-9fe6-674e69ea840e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-0a6cca-6599d4945f-dlflj" Oct 31 01:09:12.219965 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:12.219885 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:12.243472 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:09:12.243405 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod293b71e1_036c_491c_b693_f7acc63bed0f.slice/crio-0edf3ba2cdcb0226dc4f2036a76ba0b28da23e551af26f1ad8a8f6de410e9536 WatchSource:0}: Error finding container 0edf3ba2cdcb0226dc4f2036a76ba0b28da23e551af26f1ad8a8f6de410e9536: Status 404 returned error can't find the container with id 0edf3ba2cdcb0226dc4f2036a76ba0b28da23e551af26f1ad8a8f6de410e9536 Oct 31 01:09:12.244535 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:12.244508 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-1b8d05-5fdbf4bd8c-lf5w4"] Oct 31 01:09:12.246412 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:12.246390 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-0a6cca-6599d4945f-dlflj" Oct 31 01:09:12.289698 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:12.289661 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-1b8d05-6f65d64999-htfv4" event={"ID":"c80bb0f7-7c51-4344-91d9-a37dc7d21a83","Type":"ContainerStarted","Data":"09c52943d34c819ac6cddf84b089da9d4bbe5ae0929136054a66be43710e2ef7"} Oct 31 01:09:12.289840 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:12.289718 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-1b8d05-6f65d64999-htfv4" event={"ID":"c80bb0f7-7c51-4344-91d9-a37dc7d21a83","Type":"ContainerStarted","Data":"a82e12f53fd7789b48fc61cb4fed8384fbd5bb69920514a6be18b7ebcd59917c"} Oct 31 01:09:12.289840 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:12.289781 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-1b8d05-6f65d64999-htfv4" Oct 31 01:09:12.291574 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:12.291542 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1b8d05-5c48d56c8c-swf9d" event={"ID":"7c3f1628-66ce-4761-80ea-e7654895d913","Type":"ContainerStarted","Data":"714f6ab78d14b43c7b0400742976aa34100b87b05b122814af1fb1b75090f16d"} Oct 31 01:09:12.291720 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:12.291595 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1b8d05-5c48d56c8c-swf9d" Oct 31 01:09:12.291720 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:12.291616 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1b8d05-5c48d56c8c-swf9d" event={"ID":"7c3f1628-66ce-4761-80ea-e7654895d913","Type":"ContainerStarted","Data":"4e2d29b911ee825cd3960936aea2135eb9e12328bd9db595f2fd693419c5fa17"} Oct 31 01:09:12.292739 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:12.292718 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1b8d05-5fdbf4bd8c-lf5w4" event={"ID":"293b71e1-036c-491c-b693-f7acc63bed0f","Type":"ContainerStarted","Data":"0edf3ba2cdcb0226dc4f2036a76ba0b28da23e551af26f1ad8a8f6de410e9536"} Oct 31 01:09:12.319995 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:12.319563 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-1b8d05-6f65d64999-htfv4" podStartSLOduration=1.319545502 podStartE2EDuration="1.319545502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:12.314635741 +0000 UTC m=+796.673358058" watchObservedRunningTime="2026-10-31 01:09:12.319545502 +0000 UTC m=+796.678267819" Oct 31 01:09:12.341704 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:12.341645 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1b8d05-5c48d56c8c-swf9d" podStartSLOduration=1.341626444 podStartE2EDuration="1.341626444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:12.338101953 +0000 UTC m=+796.696824296" watchObservedRunningTime="2026-10-31 01:09:12.341626444 +0000 UTC m=+796.700348763" Oct 31 01:09:12.369354 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:12.369322 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:12.537446 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:12.537349 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-0a6cca-6599d4945f-dlflj"] Oct 31 01:09:12.972684 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:12.972646 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-acdaed-7ccf869bd7-ldg5t"] Oct 31 01:09:12.977356 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:12.977326 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-acdaed-7ccf869bd7-ldg5t" Oct 31 01:09:12.994117 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:12.994084 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-acdaed-serving-cert\"" Oct 31 01:09:13.005713 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:13.005679 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-acdaed-7ccf869bd7-ldg5t"] Oct 31 01:09:13.073271 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:13.073228 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-592cg\" (UniqueName: \"kubernetes.io/projected/0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3-kube-api-access-592cg\") pod \"maas-api-e2e-worker-w0-acdaed-7ccf869bd7-ldg5t\" (UID: \"0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-acdaed-7ccf869bd7-ldg5t" Oct 31 01:09:13.073449 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:13.073287 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3-maas-api-tls\") pod \"maas-api-e2e-worker-w0-acdaed-7ccf869bd7-ldg5t\" (UID: \"0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-acdaed-7ccf869bd7-ldg5t" Oct 31 01:09:13.073506 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:13.073414 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-acdaed-7ccf869bd7-ldg5t\" (UID: \"0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-acdaed-7ccf869bd7-ldg5t" Oct 31 01:09:13.073551 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:13.073506 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3-metrics-certs\") pod \"maas-api-e2e-worker-w0-acdaed-7ccf869bd7-ldg5t\" (UID: \"0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-acdaed-7ccf869bd7-ldg5t" Oct 31 01:09:13.077329 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:13.077294 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:13.174456 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:13.174409 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-592cg\" (UniqueName: \"kubernetes.io/projected/0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3-kube-api-access-592cg\") pod \"maas-api-e2e-worker-w0-acdaed-7ccf869bd7-ldg5t\" (UID: \"0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-acdaed-7ccf869bd7-ldg5t" Oct 31 01:09:13.174646 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:13.174485 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3-maas-api-tls\") pod \"maas-api-e2e-worker-w0-acdaed-7ccf869bd7-ldg5t\" (UID: \"0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-acdaed-7ccf869bd7-ldg5t" Oct 31 01:09:13.174646 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:13.174540 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-acdaed-7ccf869bd7-ldg5t\" (UID: \"0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-acdaed-7ccf869bd7-ldg5t" Oct 31 01:09:13.174761 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:13.174687 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3-metrics-certs\") pod \"maas-api-e2e-worker-w0-acdaed-7ccf869bd7-ldg5t\" (UID: \"0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-acdaed-7ccf869bd7-ldg5t" Oct 31 01:09:13.175255 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:13.175232 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-acdaed-7ccf869bd7-ldg5t\" (UID: \"0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-acdaed-7ccf869bd7-ldg5t" Oct 31 01:09:13.177062 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:13.177042 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3-maas-api-tls\") pod \"maas-api-e2e-worker-w0-acdaed-7ccf869bd7-ldg5t\" (UID: \"0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-acdaed-7ccf869bd7-ldg5t" Oct 31 01:09:13.177158 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:13.177112 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3-metrics-certs\") pod \"maas-api-e2e-worker-w0-acdaed-7ccf869bd7-ldg5t\" (UID: \"0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-acdaed-7ccf869bd7-ldg5t" Oct 31 01:09:13.181938 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:13.181917 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-592cg\" (UniqueName: \"kubernetes.io/projected/0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3-kube-api-access-592cg\") pod \"maas-api-e2e-worker-w0-acdaed-7ccf869bd7-ldg5t\" (UID: \"0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-acdaed-7ccf869bd7-ldg5t" Oct 31 01:09:13.192285 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:13.192259 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:13.295503 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:13.295127 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-acdaed-7ccf869bd7-ldg5t" Oct 31 01:09:13.319740 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:13.319708 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:13.323315 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:13.323281 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1b8d05-5fdbf4bd8c-lf5w4" event={"ID":"293b71e1-036c-491c-b693-f7acc63bed0f","Type":"ContainerStarted","Data":"63c1d7b949c20ade4dc507c1aed0c655135f40c64f0952f91fddc1c91dabc118"} Oct 31 01:09:13.324180 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:13.324144 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1b8d05-5fdbf4bd8c-lf5w4" Oct 31 01:09:13.332656 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:13.331925 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-0a6cca-6599d4945f-dlflj" event={"ID":"eaad3e7d-1293-43d5-9fe6-674e69ea840e","Type":"ContainerStarted","Data":"326d064c31cabc994f3860adf0c6c71f24c1ae6bdcd13f040b76733cd4c943c3"} Oct 31 01:09:13.332656 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:13.331977 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-0a6cca-6599d4945f-dlflj" event={"ID":"eaad3e7d-1293-43d5-9fe6-674e69ea840e","Type":"ContainerStarted","Data":"366506dbe164a835a4f265187b14df634d3cad19f2bbe1368078ceb225e71772"} Oct 31 01:09:13.332656 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:13.332044 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-0a6cca-6599d4945f-dlflj" Oct 31 01:09:13.342754 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:13.342688 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1b8d05-5fdbf4bd8c-lf5w4" podStartSLOduration=1.896562042 podStartE2EDuration="2.342671197s" totalImagesPullingTime="446.109155ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:13.340992587 +0000 UTC m=+797.699714906" watchObservedRunningTime="2026-10-31 01:09:13.342671197 +0000 UTC m=+797.701393514" Oct 31 01:09:13.361302 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:13.361254 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-0a6cca-6599d4945f-dlflj" podStartSLOduration=2.361237795 podStartE2EDuration="2.361237795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:13.358730825 +0000 UTC m=+797.717453142" watchObservedRunningTime="2026-10-31 01:09:13.361237795 +0000 UTC m=+797.719960115" Oct 31 01:09:13.622738 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:13.622666 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-acdaed-7ccf869bd7-ldg5t"] Oct 31 01:09:13.625882 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:09:13.624909 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f5d3e96_6c6e_4b38_b3e0_ce2e3db70fa3.slice/crio-f570bdf9e1c1963f218b087582ed41d70478dd859634cedd5e35e98a231c9e1c WatchSource:0}: Error finding container f570bdf9e1c1963f218b087582ed41d70478dd859634cedd5e35e98a231c9e1c: Status 404 returned error can't find the container with id f570bdf9e1c1963f218b087582ed41d70478dd859634cedd5e35e98a231c9e1c Oct 31 01:09:13.685871 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:13.685843 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:13.687779 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:13.687753 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-acdaed-55478d5599-md22s"] Oct 31 01:09:13.692773 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:13.692756 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-acdaed-55478d5599-md22s" Oct 31 01:09:13.708383 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:13.708360 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-acdaed-55478d5599-md22s"] Oct 31 01:09:13.762865 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:13.762834 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-acdaed\"" Oct 31 01:09:13.782473 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:13.782441 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfqjb\" (UniqueName: \"kubernetes.io/projected/e4160358-c655-4a6b-bb39-fc8677804241-kube-api-access-hfqjb\") pod \"payload-pre-processing-e2e-worker-w0-acdaed-55478d5599-md22s\" (UID: \"e4160358-c655-4a6b-bb39-fc8677804241\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-acdaed-55478d5599-md22s" Oct 31 01:09:13.782614 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:13.782596 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4160358-c655-4a6b-bb39-fc8677804241-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-acdaed-55478d5599-md22s\" (UID: \"e4160358-c655-4a6b-bb39-fc8677804241\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-acdaed-55478d5599-md22s" Oct 31 01:09:13.810573 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:13.810534 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:13.884035 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:13.883967 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4160358-c655-4a6b-bb39-fc8677804241-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-acdaed-55478d5599-md22s\" (UID: \"e4160358-c655-4a6b-bb39-fc8677804241\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-acdaed-55478d5599-md22s" Oct 31 01:09:13.884154 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:13.884048 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfqjb\" (UniqueName: \"kubernetes.io/projected/e4160358-c655-4a6b-bb39-fc8677804241-kube-api-access-hfqjb\") pod \"payload-pre-processing-e2e-worker-w0-acdaed-55478d5599-md22s\" (UID: \"e4160358-c655-4a6b-bb39-fc8677804241\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-acdaed-55478d5599-md22s" Oct 31 01:09:13.884809 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:13.884790 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4160358-c655-4a6b-bb39-fc8677804241-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-acdaed-55478d5599-md22s\" (UID: \"e4160358-c655-4a6b-bb39-fc8677804241\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-acdaed-55478d5599-md22s" Oct 31 01:09:13.893872 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:13.893842 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfqjb\" (UniqueName: \"kubernetes.io/projected/e4160358-c655-4a6b-bb39-fc8677804241-kube-api-access-hfqjb\") pod \"payload-pre-processing-e2e-worker-w0-acdaed-55478d5599-md22s\" (UID: \"e4160358-c655-4a6b-bb39-fc8677804241\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-acdaed-55478d5599-md22s" Oct 31 01:09:14.008489 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:14.008456 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-acdaed-55478d5599-md22s" Oct 31 01:09:14.086348 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:14.085446 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:14.199562 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:14.199509 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:14.300121 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:14.300085 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-acdaed-55478d5599-md22s"] Oct 31 01:09:14.304839 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:09:14.304806 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4160358_c655_4a6b_bb39_fc8677804241.slice/crio-bc6f9059d5390a5f1f07241e8769950da7afb9c6aeaccab20b2edc31aa17b3d7 WatchSource:0}: Error finding container bc6f9059d5390a5f1f07241e8769950da7afb9c6aeaccab20b2edc31aa17b3d7: Status 404 returned error can't find the container with id bc6f9059d5390a5f1f07241e8769950da7afb9c6aeaccab20b2edc31aa17b3d7 Oct 31 01:09:14.333709 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:14.333532 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:14.340541 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:14.340513 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-acdaed-7ccf869bd7-ldg5t" event={"ID":"0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3","Type":"ContainerStarted","Data":"7b9979ea3a8417f6f75a18b6991117c96c64ce869fee9633f9506a84ca22bf47"} Oct 31 01:09:14.340660 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:14.340577 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-acdaed-7ccf869bd7-ldg5t" Oct 31 01:09:14.340660 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:14.340604 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-acdaed-7ccf869bd7-ldg5t" event={"ID":"0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3","Type":"ContainerStarted","Data":"f570bdf9e1c1963f218b087582ed41d70478dd859634cedd5e35e98a231c9e1c"} Oct 31 01:09:14.341869 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:14.341842 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-acdaed-55478d5599-md22s" event={"ID":"e4160358-c655-4a6b-bb39-fc8677804241","Type":"ContainerStarted","Data":"bc6f9059d5390a5f1f07241e8769950da7afb9c6aeaccab20b2edc31aa17b3d7"} Oct 31 01:09:14.361183 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:14.361126 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-acdaed-7ccf869bd7-ldg5t" podStartSLOduration=1.9349286719999998 podStartE2EDuration="2.361109122s" totalImagesPullingTime="426.18045ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:14.35791259 +0000 UTC m=+798.716634910" watchObservedRunningTime="2026-10-31 01:09:14.361109122 +0000 UTC m=+798.719831443" Oct 31 01:09:14.420752 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:14.420720 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-487a6c-6f784564bc-c6qb5"] Oct 31 01:09:14.424458 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:14.424442 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-487a6c-6f784564bc-c6qb5" Oct 31 01:09:14.434069 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:14.434044 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-487a6c-6f784564bc-c6qb5"] Oct 31 01:09:14.453009 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:14.452974 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:14.489903 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:14.489869 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a3e8a39-48ca-4063-93b8-8afa2f1f18ac-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-487a6c-6f784564bc-c6qb5\" (UID: \"0a3e8a39-48ca-4063-93b8-8afa2f1f18ac\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-487a6c-6f784564bc-c6qb5" Oct 31 01:09:14.490053 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:14.489911 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcvrc\" (UniqueName: \"kubernetes.io/projected/0a3e8a39-48ca-4063-93b8-8afa2f1f18ac-kube-api-access-qcvrc\") pod \"payload-processing-e2e-worker-w3-487a6c-6f784564bc-c6qb5\" (UID: \"0a3e8a39-48ca-4063-93b8-8afa2f1f18ac\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-487a6c-6f784564bc-c6qb5" Oct 31 01:09:14.490647 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:14.490529 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-487a6c\"" Oct 31 01:09:14.490647 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:14.490568 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-487a6c-dockercfg-82tzt\"" Oct 31 01:09:14.591849 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:14.591813 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a3e8a39-48ca-4063-93b8-8afa2f1f18ac-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-487a6c-6f784564bc-c6qb5\" (UID: \"0a3e8a39-48ca-4063-93b8-8afa2f1f18ac\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-487a6c-6f784564bc-c6qb5" Oct 31 01:09:14.591849 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:14.591850 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcvrc\" (UniqueName: \"kubernetes.io/projected/0a3e8a39-48ca-4063-93b8-8afa2f1f18ac-kube-api-access-qcvrc\") pod \"payload-processing-e2e-worker-w3-487a6c-6f784564bc-c6qb5\" (UID: \"0a3e8a39-48ca-4063-93b8-8afa2f1f18ac\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-487a6c-6f784564bc-c6qb5" Oct 31 01:09:14.592621 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:14.592599 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a3e8a39-48ca-4063-93b8-8afa2f1f18ac-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-487a6c-6f784564bc-c6qb5\" (UID: \"0a3e8a39-48ca-4063-93b8-8afa2f1f18ac\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-487a6c-6f784564bc-c6qb5" Oct 31 01:09:14.599870 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:14.599837 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcvrc\" (UniqueName: \"kubernetes.io/projected/0a3e8a39-48ca-4063-93b8-8afa2f1f18ac-kube-api-access-qcvrc\") pod \"payload-processing-e2e-worker-w3-487a6c-6f784564bc-c6qb5\" (UID: \"0a3e8a39-48ca-4063-93b8-8afa2f1f18ac\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-487a6c-6f784564bc-c6qb5" Oct 31 01:09:14.736289 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:14.736032 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:14.742657 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:14.742625 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-487a6c-6f784564bc-c6qb5" Oct 31 01:09:14.829822 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:14.829789 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-28ff08-8947d86b5-frcjf"] Oct 31 01:09:14.836740 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:14.836716 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-28ff08-8947d86b5-frcjf" Oct 31 01:09:14.850949 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:14.850926 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-28ff08-8947d86b5-frcjf"] Oct 31 01:09:14.855110 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:14.855035 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:14.897503 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:14.897481 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-28ff08-dockercfg-gnbpc\"" Oct 31 01:09:14.897711 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:14.897582 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-28ff08\"" Oct 31 01:09:14.969934 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:14.969841 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:14.999500 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:14.999221 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8df1e49-cfaf-4c8c-9498-3400501ae86b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-28ff08-8947d86b5-frcjf\" (UID: \"b8df1e49-cfaf-4c8c-9498-3400501ae86b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-28ff08-8947d86b5-frcjf" Oct 31 01:09:14.999500 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:14.999271 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2bvc\" (UniqueName: \"kubernetes.io/projected/b8df1e49-cfaf-4c8c-9498-3400501ae86b-kube-api-access-x2bvc\") pod \"payload-processing-e2e-worker-w5-28ff08-8947d86b5-frcjf\" (UID: \"b8df1e49-cfaf-4c8c-9498-3400501ae86b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-28ff08-8947d86b5-frcjf" Oct 31 01:09:15.033975 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:15.033949 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-487a6c-6f784564bc-c6qb5"] Oct 31 01:09:15.037989 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:09:15.037934 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a3e8a39_48ca_4063_93b8_8afa2f1f18ac.slice/crio-74a76aa300f01e1b738e0fc3c46a6dce5cdf1c9562d4cfc6ccd433c36e097cbf WatchSource:0}: Error finding container 74a76aa300f01e1b738e0fc3c46a6dce5cdf1c9562d4cfc6ccd433c36e097cbf: Status 404 returned error can't find the container with id 74a76aa300f01e1b738e0fc3c46a6dce5cdf1c9562d4cfc6ccd433c36e097cbf Oct 31 01:09:15.040995 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:15.040968 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:09:15.100500 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:15.100474 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8df1e49-cfaf-4c8c-9498-3400501ae86b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-28ff08-8947d86b5-frcjf\" (UID: \"b8df1e49-cfaf-4c8c-9498-3400501ae86b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-28ff08-8947d86b5-frcjf" Oct 31 01:09:15.100616 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:15.100510 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2bvc\" (UniqueName: \"kubernetes.io/projected/b8df1e49-cfaf-4c8c-9498-3400501ae86b-kube-api-access-x2bvc\") pod \"payload-processing-e2e-worker-w5-28ff08-8947d86b5-frcjf\" (UID: \"b8df1e49-cfaf-4c8c-9498-3400501ae86b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-28ff08-8947d86b5-frcjf" Oct 31 01:09:15.101249 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:15.101225 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8df1e49-cfaf-4c8c-9498-3400501ae86b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-28ff08-8947d86b5-frcjf\" (UID: \"b8df1e49-cfaf-4c8c-9498-3400501ae86b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-28ff08-8947d86b5-frcjf" Oct 31 01:09:15.114157 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:15.114131 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2bvc\" (UniqueName: \"kubernetes.io/projected/b8df1e49-cfaf-4c8c-9498-3400501ae86b-kube-api-access-x2bvc\") pod \"payload-processing-e2e-worker-w5-28ff08-8947d86b5-frcjf\" (UID: \"b8df1e49-cfaf-4c8c-9498-3400501ae86b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-28ff08-8947d86b5-frcjf" Oct 31 01:09:15.167440 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:15.160639 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-28ff08-8947d86b5-frcjf" Oct 31 01:09:15.351560 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:15.350734 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-acdaed-55478d5599-md22s" event={"ID":"e4160358-c655-4a6b-bb39-fc8677804241","Type":"ContainerStarted","Data":"30fcda6740c24446c9a353da5eaecabbf6486015e8b17082862fbd1e1421fca2"} Oct 31 01:09:15.351560 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:15.350894 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-acdaed-55478d5599-md22s" Oct 31 01:09:15.353381 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:15.352717 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-487a6c-6f784564bc-c6qb5" event={"ID":"0a3e8a39-48ca-4063-93b8-8afa2f1f18ac","Type":"ContainerStarted","Data":"03b0b7a38b56730c846fed5a5f497ad2d1a8b593d453e4e09eb61e8e5182d098"} Oct 31 01:09:15.353381 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:15.352996 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-487a6c-6f784564bc-c6qb5" Oct 31 01:09:15.353381 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:15.353034 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-487a6c-6f784564bc-c6qb5" event={"ID":"0a3e8a39-48ca-4063-93b8-8afa2f1f18ac","Type":"ContainerStarted","Data":"74a76aa300f01e1b738e0fc3c46a6dce5cdf1c9562d4cfc6ccd433c36e097cbf"} Oct 31 01:09:15.389084 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:15.389032 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-acdaed-55478d5599-md22s" podStartSLOduration=2.389019893 podStartE2EDuration="2.389019893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:15.385488231 +0000 UTC m=+799.744210548" watchObservedRunningTime="2026-10-31 01:09:15.389019893 +0000 UTC m=+799.747742209" Oct 31 01:09:15.424756 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:15.423917 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-487a6c-6f784564bc-c6qb5" podStartSLOduration=1.423900398 podStartE2EDuration="1.423900398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:15.417741276 +0000 UTC m=+799.776463602" watchObservedRunningTime="2026-10-31 01:09:15.423900398 +0000 UTC m=+799.782622714" Oct 31 01:09:15.534879 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:15.534849 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-28ff08-8947d86b5-frcjf"] Oct 31 01:09:15.535348 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:09:15.535322 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8df1e49_cfaf_4c8c_9498_3400501ae86b.slice/crio-fcb90366b81fcd3c25060076a4a89225f9444e67bffa4ebb0934a2971844e628 WatchSource:0}: Error finding container fcb90366b81fcd3c25060076a4a89225f9444e67bffa4ebb0934a2971844e628: Status 404 returned error can't find the container with id fcb90366b81fcd3c25060076a4a89225f9444e67bffa4ebb0934a2971844e628 Oct 31 01:09:15.573805 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:15.573775 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:15.617090 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:15.617064 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8499c8-5c65ddd74c-9r5n8"] Oct 31 01:09:15.620890 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:15.620866 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8499c8-5c65ddd74c-9r5n8" Oct 31 01:09:15.630157 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:15.630121 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-8499c8\"" Oct 31 01:09:15.630302 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:15.630286 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-8499c8-dockercfg-gcnzp\"" Oct 31 01:09:15.631715 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:15.631695 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8499c8-5c65ddd74c-9r5n8"] Oct 31 01:09:15.697819 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:15.697777 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:15.717448 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:15.714013 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab8ba016-584a-4646-a3d4-f6db4ccf0700-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-8499c8-5c65ddd74c-9r5n8\" (UID: \"ab8ba016-584a-4646-a3d4-f6db4ccf0700\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8499c8-5c65ddd74c-9r5n8" Oct 31 01:09:15.750489 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:15.744008 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr2c5\" (UniqueName: \"kubernetes.io/projected/ab8ba016-584a-4646-a3d4-f6db4ccf0700-kube-api-access-xr2c5\") pod \"payload-pre-processing-e2e-worker-w4-8499c8-5c65ddd74c-9r5n8\" (UID: \"ab8ba016-584a-4646-a3d4-f6db4ccf0700\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8499c8-5c65ddd74c-9r5n8" Oct 31 01:09:15.851951 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:15.851855 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab8ba016-584a-4646-a3d4-f6db4ccf0700-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-8499c8-5c65ddd74c-9r5n8\" (UID: \"ab8ba016-584a-4646-a3d4-f6db4ccf0700\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8499c8-5c65ddd74c-9r5n8" Oct 31 01:09:15.852138 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:15.851962 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr2c5\" (UniqueName: \"kubernetes.io/projected/ab8ba016-584a-4646-a3d4-f6db4ccf0700-kube-api-access-xr2c5\") pod \"payload-pre-processing-e2e-worker-w4-8499c8-5c65ddd74c-9r5n8\" (UID: \"ab8ba016-584a-4646-a3d4-f6db4ccf0700\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8499c8-5c65ddd74c-9r5n8" Oct 31 01:09:15.852722 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:15.852694 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab8ba016-584a-4646-a3d4-f6db4ccf0700-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-8499c8-5c65ddd74c-9r5n8\" (UID: \"ab8ba016-584a-4646-a3d4-f6db4ccf0700\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8499c8-5c65ddd74c-9r5n8" Oct 31 01:09:15.859965 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:15.859942 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr2c5\" (UniqueName: \"kubernetes.io/projected/ab8ba016-584a-4646-a3d4-f6db4ccf0700-kube-api-access-xr2c5\") pod \"payload-pre-processing-e2e-worker-w4-8499c8-5c65ddd74c-9r5n8\" (UID: \"ab8ba016-584a-4646-a3d4-f6db4ccf0700\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8499c8-5c65ddd74c-9r5n8" Oct 31 01:09:15.938630 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:15.938599 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8499c8-5c65ddd74c-9r5n8" Oct 31 01:09:15.981385 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:15.981356 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:16.140713 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:16.140618 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:16.250672 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:16.250632 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8499c8-5c65ddd74c-9r5n8"] Oct 31 01:09:16.257539 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:16.257504 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:16.366448 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:16.365900 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-28ff08-8947d86b5-frcjf" event={"ID":"b8df1e49-cfaf-4c8c-9498-3400501ae86b","Type":"ContainerStarted","Data":"132e471216ed1fe2da347dc8ccaf1379ba6bac80ff6fd23338cbd50509a26a0c"} Oct 31 01:09:16.366448 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:16.365972 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-28ff08-8947d86b5-frcjf" event={"ID":"b8df1e49-cfaf-4c8c-9498-3400501ae86b","Type":"ContainerStarted","Data":"fcb90366b81fcd3c25060076a4a89225f9444e67bffa4ebb0934a2971844e628"} Oct 31 01:09:16.366687 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:16.366518 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-28ff08-8947d86b5-frcjf" Oct 31 01:09:16.367941 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:16.367911 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8499c8-5c65ddd74c-9r5n8" event={"ID":"ab8ba016-584a-4646-a3d4-f6db4ccf0700","Type":"ContainerStarted","Data":"fdc8c75c7337e509923238f84ca0a311a7d46943673d423c4bb2de5b393c686e"} Oct 31 01:09:16.393627 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:16.393551 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-28ff08-8947d86b5-frcjf" podStartSLOduration=2.393536595 podStartE2EDuration="2.393536595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:16.389866706 +0000 UTC m=+800.748589023" watchObservedRunningTime="2026-10-31 01:09:16.393536595 +0000 UTC m=+800.752258912" Oct 31 01:09:16.747013 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:16.746972 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:17.380746 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:17.380697 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8499c8-5c65ddd74c-9r5n8" event={"ID":"ab8ba016-584a-4646-a3d4-f6db4ccf0700","Type":"ContainerStarted","Data":"ba5565b2f2730d579c090408859c176f458b63d24c3adf03db39fe85462fe15e"} Oct 31 01:09:17.423595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:17.423532 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8499c8-5c65ddd74c-9r5n8" podStartSLOduration=2.423512687 podStartE2EDuration="2.423512687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:17.400649749 +0000 UTC m=+801.759372066" watchObservedRunningTime="2026-10-31 01:09:17.423512687 +0000 UTC m=+801.782235005" Oct 31 01:09:17.562773 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:17.562737 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-7f6d1e-5754ff484f-dzhmh"] Oct 31 01:09:17.569493 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:17.569394 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-7f6d1e-5754ff484f-dzhmh"] Oct 31 01:09:17.569644 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:17.569555 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7f6d1e-5754ff484f-dzhmh" Oct 31 01:09:17.573554 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:17.573352 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-7f6d1e-serving-cert\"" Oct 31 01:09:17.595180 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:17.595083 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-7f6d1e-5754ff484f-dzhmh\" (UID: \"f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7f6d1e-5754ff484f-dzhmh" Oct 31 01:09:17.595340 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:17.595324 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60-maas-api-tls\") pod \"maas-api-e2e-worker-w1-7f6d1e-5754ff484f-dzhmh\" (UID: \"f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7f6d1e-5754ff484f-dzhmh" Oct 31 01:09:17.595394 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:17.595361 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60-metrics-certs\") pod \"maas-api-e2e-worker-w1-7f6d1e-5754ff484f-dzhmh\" (UID: \"f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7f6d1e-5754ff484f-dzhmh" Oct 31 01:09:17.634406 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:17.615551 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94fzl\" (UniqueName: \"kubernetes.io/projected/f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60-kube-api-access-94fzl\") pod \"maas-api-e2e-worker-w1-7f6d1e-5754ff484f-dzhmh\" (UID: \"f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7f6d1e-5754ff484f-dzhmh" Oct 31 01:09:17.634406 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:17.621001 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:17.729443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:17.729006 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60-maas-api-tls\") pod \"maas-api-e2e-worker-w1-7f6d1e-5754ff484f-dzhmh\" (UID: \"f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7f6d1e-5754ff484f-dzhmh" Oct 31 01:09:17.729443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:17.729064 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60-metrics-certs\") pod \"maas-api-e2e-worker-w1-7f6d1e-5754ff484f-dzhmh\" (UID: \"f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7f6d1e-5754ff484f-dzhmh" Oct 31 01:09:17.729443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:17.729097 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94fzl\" (UniqueName: \"kubernetes.io/projected/f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60-kube-api-access-94fzl\") pod \"maas-api-e2e-worker-w1-7f6d1e-5754ff484f-dzhmh\" (UID: \"f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7f6d1e-5754ff484f-dzhmh" Oct 31 01:09:17.729443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:17.729211 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-7f6d1e-5754ff484f-dzhmh\" (UID: \"f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7f6d1e-5754ff484f-dzhmh" Oct 31 01:09:17.743450 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:17.730136 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-7f6d1e-5754ff484f-dzhmh\" (UID: \"f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7f6d1e-5754ff484f-dzhmh" Oct 31 01:09:17.743450 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:17.736753 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:17.756466 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:17.755984 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60-metrics-certs\") pod \"maas-api-e2e-worker-w1-7f6d1e-5754ff484f-dzhmh\" (UID: \"f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7f6d1e-5754ff484f-dzhmh" Oct 31 01:09:17.757242 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:17.756852 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94fzl\" (UniqueName: \"kubernetes.io/projected/f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60-kube-api-access-94fzl\") pod \"maas-api-e2e-worker-w1-7f6d1e-5754ff484f-dzhmh\" (UID: \"f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7f6d1e-5754ff484f-dzhmh" Oct 31 01:09:17.759851 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:17.758902 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60-maas-api-tls\") pod \"maas-api-e2e-worker-w1-7f6d1e-5754ff484f-dzhmh\" (UID: \"f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7f6d1e-5754ff484f-dzhmh" Oct 31 01:09:17.852205 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:17.852145 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:17.885291 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:17.885210 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7f6d1e-5754ff484f-dzhmh" Oct 31 01:09:18.385670 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:18.384853 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:18.397707 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:18.397671 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-7f6d1e-5754ff484f-dzhmh"] Oct 31 01:09:18.462181 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:18.462144 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7f6d1e-5754ff484f-dzhmh" event={"ID":"f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60","Type":"ContainerStarted","Data":"69a42b964f73585ef31eeed5cbf3c554703c4ba57b33c8c88efe14bea872c759"} Oct 31 01:09:18.462770 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:18.462746 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8499c8-5c65ddd74c-9r5n8" Oct 31 01:09:18.489610 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:18.489570 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:18.616505 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:18.613957 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:19.056439 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:19.033524 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj"] Oct 31 01:09:19.077526 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:19.077503 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj"] Oct 31 01:09:19.077783 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:19.077768 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj" Oct 31 01:09:19.081330 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:19.081298 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-8499c8-serving-cert\"" Oct 31 01:09:19.084128 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:19.084100 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:19.193438 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:19.192359 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:19.220443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:19.212355 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25r9x\" (UniqueName: \"kubernetes.io/projected/be67bf17-2e3c-4624-96cd-aab05f0cdc27-kube-api-access-25r9x\") pod \"maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj\" (UID: \"be67bf17-2e3c-4624-96cd-aab05f0cdc27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj" Oct 31 01:09:19.220443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:19.212437 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be67bf17-2e3c-4624-96cd-aab05f0cdc27-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj\" (UID: \"be67bf17-2e3c-4624-96cd-aab05f0cdc27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj" Oct 31 01:09:19.220443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:19.212519 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be67bf17-2e3c-4624-96cd-aab05f0cdc27-metrics-certs\") pod \"maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj\" (UID: \"be67bf17-2e3c-4624-96cd-aab05f0cdc27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj" Oct 31 01:09:19.220443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:19.212582 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be67bf17-2e3c-4624-96cd-aab05f0cdc27-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj\" (UID: \"be67bf17-2e3c-4624-96cd-aab05f0cdc27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj" Oct 31 01:09:19.291026 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:19.290981 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:19.316851 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:19.316027 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be67bf17-2e3c-4624-96cd-aab05f0cdc27-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj\" (UID: \"be67bf17-2e3c-4624-96cd-aab05f0cdc27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj" Oct 31 01:09:19.316851 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:19.316138 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25r9x\" (UniqueName: \"kubernetes.io/projected/be67bf17-2e3c-4624-96cd-aab05f0cdc27-kube-api-access-25r9x\") pod \"maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj\" (UID: \"be67bf17-2e3c-4624-96cd-aab05f0cdc27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj" Oct 31 01:09:19.316851 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:19.316183 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be67bf17-2e3c-4624-96cd-aab05f0cdc27-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj\" (UID: \"be67bf17-2e3c-4624-96cd-aab05f0cdc27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj" Oct 31 01:09:19.316851 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:19.316252 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be67bf17-2e3c-4624-96cd-aab05f0cdc27-metrics-certs\") pod \"maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj\" (UID: \"be67bf17-2e3c-4624-96cd-aab05f0cdc27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj" Oct 31 01:09:19.339446 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:19.319154 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be67bf17-2e3c-4624-96cd-aab05f0cdc27-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj\" (UID: \"be67bf17-2e3c-4624-96cd-aab05f0cdc27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj" Oct 31 01:09:19.352440 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:19.345989 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25r9x\" (UniqueName: \"kubernetes.io/projected/be67bf17-2e3c-4624-96cd-aab05f0cdc27-kube-api-access-25r9x\") pod \"maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj\" (UID: \"be67bf17-2e3c-4624-96cd-aab05f0cdc27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj" Oct 31 01:09:19.352440 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:19.351833 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be67bf17-2e3c-4624-96cd-aab05f0cdc27-metrics-certs\") pod \"maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj\" (UID: \"be67bf17-2e3c-4624-96cd-aab05f0cdc27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj" Oct 31 01:09:19.359444 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:19.359316 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be67bf17-2e3c-4624-96cd-aab05f0cdc27-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj\" (UID: \"be67bf17-2e3c-4624-96cd-aab05f0cdc27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj" Oct 31 01:09:19.442571 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:19.433630 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj" Oct 31 01:09:19.472976 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:19.472946 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7f6d1e-5754ff484f-dzhmh" event={"ID":"f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60","Type":"ContainerStarted","Data":"a90d9f358e7f3f9de81333b9930a17651b2baabea3bfed1f37140f8a19e6e2db"} Oct 31 01:09:19.473479 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:19.473361 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7f6d1e-5754ff484f-dzhmh" Oct 31 01:09:19.514442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:19.497356 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7f6d1e-5754ff484f-dzhmh" podStartSLOduration=2.030405343 podStartE2EDuration="2.497337555s" totalImagesPullingTime="466.932212ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:19.492316413 +0000 UTC m=+803.851038729" watchObservedRunningTime="2026-10-31 01:09:19.497337555 +0000 UTC m=+803.856059872" Oct 31 01:09:19.827502 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:19.825232 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj"] Oct 31 01:09:19.908537 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:19.908473 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:20.366159 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:20.366129 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1b8d05-5fdbf4bd8c-lf5w4" Oct 31 01:09:20.367405 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:20.367373 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-acdaed-7ccf869bd7-ldg5t" Oct 31 01:09:20.574741 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:20.574229 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj" event={"ID":"be67bf17-2e3c-4624-96cd-aab05f0cdc27","Type":"ContainerStarted","Data":"023668299ce4f37e209367d12bd324d86b051604d890f81ce338f38cee8da3c1"} Oct 31 01:09:20.688854 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:20.688775 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:21.583522 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:21.583486 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj" event={"ID":"be67bf17-2e3c-4624-96cd-aab05f0cdc27","Type":"ContainerStarted","Data":"cf5a35c93de6f2c77d63bd35bd61db926d573cc99f29fb531007f0d3f9fb65ae"} Oct 31 01:09:21.583959 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:21.583620 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj" Oct 31 01:09:21.615442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:21.612482 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:21.632885 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:21.632811 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj" podStartSLOduration=3.098414515 podStartE2EDuration="3.63278681s" totalImagesPullingTime="534.372295ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:21.606276417 +0000 UTC m=+805.964998735" watchObservedRunningTime="2026-10-31 01:09:21.63278681 +0000 UTC m=+805.991509128" Oct 31 01:09:22.478850 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:22.478808 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:23.116262 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:23.116162 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:23.335080 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:23.335040 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1b8d05-5c48d56c8c-swf9d" Oct 31 01:09:23.335688 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:23.335662 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-1b8d05-6f65d64999-htfv4" Oct 31 01:09:23.947929 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:23.947895 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:24.345879 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:24.345791 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-0a6cca-6599d4945f-dlflj" Oct 31 01:09:24.598585 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:24.598493 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:24.716356 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:24.716310 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:25.224454 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:25.224400 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:25.303633 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:25.303593 4440 patch_prober.go:29] interesting pod/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:09:25.303882 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:25.303853 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" podUID="81a46413-c078-447a-9bbb-d9ea9431469e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:09:25.312490 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:25.312452 4440 patch_prober.go:29] interesting pod/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:09:25.312738 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:25.312711 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" podUID="cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:09:25.343295 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:25.343259 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:25.455979 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:25.455939 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:25.579945 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:25.579917 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7f6d1e-5754ff484f-dzhmh" Oct 31 01:09:25.617960 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:25.617921 4440 generic.go:356] "Generic (PLEG): container finished" podID="cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf" containerID="c26e3bab275a94d05499b10f9528f1f51aa858a8dba45c8ba18710f2b1c4811a" exitCode=0 Oct 31 01:09:25.618161 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:25.618115 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" event={"ID":"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf","Type":"ContainerDied","Data":"c26e3bab275a94d05499b10f9528f1f51aa858a8dba45c8ba18710f2b1c4811a"} Oct 31 01:09:25.618644 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:25.618621 4440 scope.go:122] "RemoveContainer" containerID="c26e3bab275a94d05499b10f9528f1f51aa858a8dba45c8ba18710f2b1c4811a" Oct 31 01:09:25.623142 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:25.620873 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:09:25.623142 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:25.620966 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:09:25.623142 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:25.622331 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:09:26.080027 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:26.079401 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:26.293930 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:26.293895 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:09:26.372184 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:26.372118 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-487a6c-6f784564bc-c6qb5" Oct 31 01:09:26.372184 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:26.372181 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-acdaed-55478d5599-md22s" Oct 31 01:09:26.577962 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:26.577926 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:26.631143 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:26.631041 4440 generic.go:356] "Generic (PLEG): container finished" podID="81a46413-c078-447a-9bbb-d9ea9431469e" containerID="cb4b0e121cee63b353d20ad49269a0abf7758c19fdb73b48fd1d8b452a7a720d" exitCode=0 Oct 31 01:09:26.631143 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:26.631125 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" event={"ID":"81a46413-c078-447a-9bbb-d9ea9431469e","Type":"ContainerDied","Data":"cb4b0e121cee63b353d20ad49269a0abf7758c19fdb73b48fd1d8b452a7a720d"} Oct 31 01:09:26.631729 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:26.631703 4440 scope.go:122] "RemoveContainer" containerID="cb4b0e121cee63b353d20ad49269a0abf7758c19fdb73b48fd1d8b452a7a720d" Oct 31 01:09:26.633372 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:26.633337 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:09:26.633516 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:26.633414 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:09:26.633586 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:26.633521 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:09:26.633949 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:26.633923 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" event={"ID":"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf","Type":"ContainerStarted","Data":"eee38b000e0b1934cca38ee3960c9cf056fac56e9bea8e3dde35d32f831b7406"} Oct 31 01:09:26.869480 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:26.863264 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:27.074148 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:27.074108 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" Oct 31 01:09:27.074379 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:27.074352 4440 patch_prober.go:29] interesting pod/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 31 01:09:27.074483 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:27.074455 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" podUID="cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 31 01:09:27.384785 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:27.384699 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-28ff08-8947d86b5-frcjf" Oct 31 01:09:27.623443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:27.622590 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj" Oct 31 01:09:27.677663 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:27.677624 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" event={"ID":"81a46413-c078-447a-9bbb-d9ea9431469e","Type":"ContainerStarted","Data":"3b795f03e8a4813ba1a35e62abc282d37d77cc108e93b21613fd53c0d1dc8e41"} Oct 31 01:09:28.080503 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:28.076949 4440 patch_prober.go:29] interesting pod/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 31 01:09:28.080503 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:28.077026 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" podUID="cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 31 01:09:28.081798 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:28.081764 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:28.669916 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:28.669877 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" Oct 31 01:09:28.674439 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:28.670678 4440 patch_prober.go:29] interesting pod/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 31 01:09:28.674439 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:28.670736 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" podUID="81a46413-c078-447a-9bbb-d9ea9431469e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 31 01:09:28.966498 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:28.966088 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:29.073913 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:29.073877 4440 patch_prober.go:29] interesting pod/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 31 01:09:29.074210 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:29.074178 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" podUID="cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 31 01:09:29.487110 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:29.487073 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8499c8-5c65ddd74c-9r5n8" Oct 31 01:09:29.655908 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:29.655865 4440 patch_prober.go:29] interesting pod/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 31 01:09:29.656200 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:29.656168 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" podUID="81a46413-c078-447a-9bbb-d9ea9431469e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 31 01:09:29.817506 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:29.813116 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:30.074507 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:30.073853 4440 patch_prober.go:29] interesting pod/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 31 01:09:30.074507 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:30.073924 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" podUID="cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 31 01:09:30.593221 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:30.593115 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:30.652968 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:30.652918 4440 patch_prober.go:29] interesting pod/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 31 01:09:30.653279 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:30.653225 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" podUID="81a46413-c078-447a-9bbb-d9ea9431469e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 31 01:09:30.883507 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:30.873213 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:31.074714 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:31.074669 4440 patch_prober.go:29] interesting pod/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 31 01:09:31.075012 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:31.074969 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" podUID="cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 31 01:09:31.118452 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:31.117891 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:31.188104 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:09:31.188068 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:09:31.554502 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:31.544149 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:31.652881 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:31.652846 4440 patch_prober.go:29] interesting pod/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 31 01:09:31.653219 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:31.653171 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" podUID="81a46413-c078-447a-9bbb-d9ea9431469e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 31 01:09:31.732927 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:31.732888 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:31.887127 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:31.887031 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:32.074320 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:32.074275 4440 patch_prober.go:29] interesting pod/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 31 01:09:32.074648 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:32.074352 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" podUID="cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 31 01:09:32.661848 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:32.661813 4440 patch_prober.go:29] interesting pod/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 31 01:09:32.662146 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:32.662113 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" podUID="81a46413-c078-447a-9bbb-d9ea9431469e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 31 01:09:32.874127 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:32.874092 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:33.054472 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:33.054437 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:33.077466 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:33.077402 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" Oct 31 01:09:33.077913 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:33.077892 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" Oct 31 01:09:33.078809 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:33.078792 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" Oct 31 01:09:33.885014 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:33.884973 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:33.932442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:33.931705 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" Oct 31 01:09:33.932442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:33.932396 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" Oct 31 01:09:33.936250 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:33.936223 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" Oct 31 01:09:34.718667 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:34.718624 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:35.126539 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:35.126445 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:35.246430 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:35.246387 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:35.536605 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:35.536559 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:35.763604 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:35.763549 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:35.903872 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:35.903781 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:36.028567 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:36.025948 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:36.292851 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:36.292808 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:36.393853 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:36.393800 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:36.512380 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:36.512345 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:09:36.530206 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:36.530156 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:36.825851 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:36.825798 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:37.116756 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:37.116663 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:37.251579 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:37.251212 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:37.648297 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:37.648169 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:38.114813 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:38.114782 4440 generic.go:356] "Generic (PLEG): container finished" podID="f8555032-1556-4f20-a9bb-7f1dcbf99dcc" containerID="d224ea1ef076b80805213e38b80de1bc55b5b9f72bf1d142e023460bd3573b22" exitCode=0 Oct 31 01:09:38.115236 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:38.114846 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" event={"ID":"f8555032-1556-4f20-a9bb-7f1dcbf99dcc","Type":"ContainerDied","Data":"d224ea1ef076b80805213e38b80de1bc55b5b9f72bf1d142e023460bd3573b22"} Oct 31 01:09:38.115283 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:38.115238 4440 scope.go:122] "RemoveContainer" containerID="d224ea1ef076b80805213e38b80de1bc55b5b9f72bf1d142e023460bd3573b22" Oct 31 01:09:38.116606 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:38.116579 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:09:38.116672 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:38.116651 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:09:38.116708 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:38.116681 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:09:38.512739 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:38.512697 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:39.128871 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:39.128835 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" event={"ID":"f8555032-1556-4f20-a9bb-7f1dcbf99dcc","Type":"ContainerStarted","Data":"829a6af953a02de438e6a74379be137cd8a51cadc3ec4a685b782715a031d660"} Oct 31 01:09:39.194354 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:39.194318 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:39.262707 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:39.262676 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" Oct 31 01:09:39.813992 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:39.813954 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:39.851731 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:39.851686 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" Oct 31 01:09:39.853095 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:39.853064 4440 patch_prober.go:29] interesting pod/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 31 01:09:39.853227 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:39.853125 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" podUID="f8555032-1556-4f20-a9bb-7f1dcbf99dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 31 01:09:40.437765 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:40.437722 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:40.549547 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:40.549256 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:40.851625 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:40.851529 4440 patch_prober.go:29] interesting pod/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 31 01:09:40.851625 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:40.851604 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" podUID="f8555032-1556-4f20-a9bb-7f1dcbf99dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 31 01:09:41.114800 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:41.114682 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:41.854997 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:41.854955 4440 patch_prober.go:29] interesting pod/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 31 01:09:41.855638 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:41.855591 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" podUID="f8555032-1556-4f20-a9bb-7f1dcbf99dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 31 01:09:41.857993 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:41.857950 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:42.065894 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:42.065859 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq"] Oct 31 01:09:42.068794 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:42.068770 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" Oct 31 01:09:42.081207 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:42.081155 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-0a6cca\"/\"default-dockercfg-2ncjr\"" Oct 31 01:09:42.081595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:42.081484 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-0a6cca\"/\"simulator-w2-0a6cca-kserve-self-signed-certs\"" Oct 31 01:09:42.085802 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:42.085759 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-0a6cca\"/\"kube-root-ca.crt\"" Oct 31 01:09:42.086035 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:42.086010 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-0a6cca\"/\"openshift-service-ca.crt\"" Oct 31 01:09:42.089658 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:42.089622 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq"] Oct 31 01:09:42.138257 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:42.138120 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:42.170217 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:42.170181 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqgtr\" (UniqueName: \"kubernetes.io/projected/4ab080a7-963c-45db-a83c-2c359b1bcf06-kube-api-access-pqgtr\") pod \"simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq\" (UID: \"4ab080a7-963c-45db-a83c-2c359b1bcf06\") " pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" Oct 31 01:09:42.170399 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:42.170266 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ab080a7-963c-45db-a83c-2c359b1bcf06-tls-certs\") pod \"simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq\" (UID: \"4ab080a7-963c-45db-a83c-2c359b1bcf06\") " pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" Oct 31 01:09:42.170399 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:42.170294 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ab080a7-963c-45db-a83c-2c359b1bcf06-home\") pod \"simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq\" (UID: \"4ab080a7-963c-45db-a83c-2c359b1bcf06\") " pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" Oct 31 01:09:42.170399 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:42.170314 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ab080a7-963c-45db-a83c-2c359b1bcf06-model-cache\") pod \"simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq\" (UID: \"4ab080a7-963c-45db-a83c-2c359b1bcf06\") " pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" Oct 31 01:09:42.170525 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:42.170412 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ab080a7-963c-45db-a83c-2c359b1bcf06-tmp-dir\") pod \"simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq\" (UID: \"4ab080a7-963c-45db-a83c-2c359b1bcf06\") " pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" Oct 31 01:09:42.170560 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:42.170522 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ab080a7-963c-45db-a83c-2c359b1bcf06-dshm\") pod \"simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq\" (UID: \"4ab080a7-963c-45db-a83c-2c359b1bcf06\") " pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" Oct 31 01:09:42.271841 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:42.271800 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqgtr\" (UniqueName: \"kubernetes.io/projected/4ab080a7-963c-45db-a83c-2c359b1bcf06-kube-api-access-pqgtr\") pod \"simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq\" (UID: \"4ab080a7-963c-45db-a83c-2c359b1bcf06\") " pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" Oct 31 01:09:42.272065 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:42.271908 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ab080a7-963c-45db-a83c-2c359b1bcf06-tls-certs\") pod \"simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq\" (UID: \"4ab080a7-963c-45db-a83c-2c359b1bcf06\") " pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" Oct 31 01:09:42.272065 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:42.271942 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ab080a7-963c-45db-a83c-2c359b1bcf06-home\") pod \"simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq\" (UID: \"4ab080a7-963c-45db-a83c-2c359b1bcf06\") " pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" Oct 31 01:09:42.272065 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:42.271969 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ab080a7-963c-45db-a83c-2c359b1bcf06-model-cache\") pod \"simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq\" (UID: \"4ab080a7-963c-45db-a83c-2c359b1bcf06\") " pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" Oct 31 01:09:42.272065 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:42.272002 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ab080a7-963c-45db-a83c-2c359b1bcf06-tmp-dir\") pod \"simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq\" (UID: \"4ab080a7-963c-45db-a83c-2c359b1bcf06\") " pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" Oct 31 01:09:42.272303 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:42.272096 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ab080a7-963c-45db-a83c-2c359b1bcf06-dshm\") pod \"simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq\" (UID: \"4ab080a7-963c-45db-a83c-2c359b1bcf06\") " pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" Oct 31 01:09:42.272534 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:42.272512 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ab080a7-963c-45db-a83c-2c359b1bcf06-model-cache\") pod \"simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq\" (UID: \"4ab080a7-963c-45db-a83c-2c359b1bcf06\") " pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" Oct 31 01:09:42.272632 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:42.272516 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ab080a7-963c-45db-a83c-2c359b1bcf06-tmp-dir\") pod \"simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq\" (UID: \"4ab080a7-963c-45db-a83c-2c359b1bcf06\") " pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" Oct 31 01:09:42.272677 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:42.272657 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ab080a7-963c-45db-a83c-2c359b1bcf06-home\") pod \"simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq\" (UID: \"4ab080a7-963c-45db-a83c-2c359b1bcf06\") " pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" Oct 31 01:09:42.274700 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:42.274677 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ab080a7-963c-45db-a83c-2c359b1bcf06-tls-certs\") pod \"simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq\" (UID: \"4ab080a7-963c-45db-a83c-2c359b1bcf06\") " pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" Oct 31 01:09:42.274811 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:42.274797 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ab080a7-963c-45db-a83c-2c359b1bcf06-dshm\") pod \"simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq\" (UID: \"4ab080a7-963c-45db-a83c-2c359b1bcf06\") " pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" Oct 31 01:09:42.282720 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:42.282690 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqgtr\" (UniqueName: \"kubernetes.io/projected/4ab080a7-963c-45db-a83c-2c359b1bcf06-kube-api-access-pqgtr\") pod \"simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq\" (UID: \"4ab080a7-963c-45db-a83c-2c359b1bcf06\") " pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" Oct 31 01:09:42.387632 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:42.387588 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" Oct 31 01:09:42.724096 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:42.724001 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:42.783161 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:42.783130 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq"] Oct 31 01:09:42.785273 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:09:42.785247 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ab080a7_963c_45db_a83c_2c359b1bcf06.slice/crio-f454110f5084afe957195d08ceb0e766f574bdb27a969d71a5f790525d99d43d WatchSource:0}: Error finding container f454110f5084afe957195d08ceb0e766f574bdb27a969d71a5f790525d99d43d: Status 404 returned error can't find the container with id f454110f5084afe957195d08ceb0e766f574bdb27a969d71a5f790525d99d43d Oct 31 01:09:42.851624 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:42.851591 4440 patch_prober.go:29] interesting pod/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 31 01:09:42.851757 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:42.851649 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" podUID="f8555032-1556-4f20-a9bb-7f1dcbf99dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 31 01:09:43.170296 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:43.170261 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" event={"ID":"4ab080a7-963c-45db-a83c-2c359b1bcf06","Type":"ContainerStarted","Data":"e87b951688d5a10a0c118c8185589d2c6ac8eb73c32557d515181916a0cc1437"} Oct 31 01:09:43.170720 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:43.170314 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" event={"ID":"4ab080a7-963c-45db-a83c-2c359b1bcf06","Type":"ContainerStarted","Data":"f454110f5084afe957195d08ceb0e766f574bdb27a969d71a5f790525d99d43d"} Oct 31 01:09:43.188545 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:43.188495 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podStartSLOduration=1.188481529 podStartE2EDuration="1.188481529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:43.186479187 +0000 UTC m=+827.545201496" watchObservedRunningTime="2026-10-31 01:09:43.188481529 +0000 UTC m=+827.547203844" Oct 31 01:09:43.388438 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:43.388391 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" Oct 31 01:09:43.494053 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:43.493950 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:43.851481 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:43.851392 4440 patch_prober.go:29] interesting pod/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 31 01:09:43.851638 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:43.851476 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" podUID="f8555032-1556-4f20-a9bb-7f1dcbf99dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 31 01:09:44.249284 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:44.249240 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:44.856271 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:44.856234 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" Oct 31 01:09:44.857296 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:44.857273 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" Oct 31 01:09:44.987556 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:44.987518 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:45.596974 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:45.596939 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:45.720882 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:45.720848 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:46.122510 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:46.122477 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:46.486346 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:46.486297 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:47.311084 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:47.311048 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:47.991021 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:47.990978 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:48.549385 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:48.549337 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:48.700989 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:48.700950 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:09:50.006832 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:50.006797 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:50.242644 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:50.242537 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:50.897003 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:50.896961 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:51.783554 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:51.783512 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:52.388771 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:52.388735 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" Oct 31 01:09:52.402384 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:52.402355 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" Oct 31 01:09:52.586689 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:52.586653 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:53.278873 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:53.278837 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" Oct 31 01:09:53.307634 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:53.307593 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:54.082900 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:54.082861 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:54.689237 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:54.689197 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:55.405117 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:55.405073 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:56.274392 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:56.274336 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:56.890799 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:56.890763 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:57.199611 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:57.199559 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk"] Oct 31 01:09:57.203749 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:57.203722 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" Oct 31 01:09:57.216734 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:57.216707 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-1b8d05\"/\"openshift-service-ca.crt\"" Oct 31 01:09:57.216868 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:57.216760 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-1b8d05\"/\"kube-root-ca.crt\"" Oct 31 01:09:57.217320 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:57.217301 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-1b8d05\"/\"premium-w6-1b8d05-kserve-self-signed-certs\"" Oct 31 01:09:57.217320 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:57.217308 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-1b8d05\"/\"default-dockercfg-r9qh5\"" Oct 31 01:09:57.223139 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:57.223112 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk"] Oct 31 01:09:57.318478 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:57.318396 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08a5f6da-8cc4-4a5a-9666-fe35c9574f95-dshm\") pod \"premium-w6-1b8d05-kserve-b68468ddd-xv5fk\" (UID: \"08a5f6da-8cc4-4a5a-9666-fe35c9574f95\") " pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" Oct 31 01:09:57.318880 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:57.318563 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n56wx\" (UniqueName: \"kubernetes.io/projected/08a5f6da-8cc4-4a5a-9666-fe35c9574f95-kube-api-access-n56wx\") pod \"premium-w6-1b8d05-kserve-b68468ddd-xv5fk\" (UID: \"08a5f6da-8cc4-4a5a-9666-fe35c9574f95\") " pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" Oct 31 01:09:57.318880 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:57.318600 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08a5f6da-8cc4-4a5a-9666-fe35c9574f95-home\") pod \"premium-w6-1b8d05-kserve-b68468ddd-xv5fk\" (UID: \"08a5f6da-8cc4-4a5a-9666-fe35c9574f95\") " pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" Oct 31 01:09:57.318880 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:57.318690 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08a5f6da-8cc4-4a5a-9666-fe35c9574f95-model-cache\") pod \"premium-w6-1b8d05-kserve-b68468ddd-xv5fk\" (UID: \"08a5f6da-8cc4-4a5a-9666-fe35c9574f95\") " pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" Oct 31 01:09:57.318880 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:57.318767 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08a5f6da-8cc4-4a5a-9666-fe35c9574f95-tls-certs\") pod \"premium-w6-1b8d05-kserve-b68468ddd-xv5fk\" (UID: \"08a5f6da-8cc4-4a5a-9666-fe35c9574f95\") " pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" Oct 31 01:09:57.318880 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:57.318805 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08a5f6da-8cc4-4a5a-9666-fe35c9574f95-tmp-dir\") pod \"premium-w6-1b8d05-kserve-b68468ddd-xv5fk\" (UID: \"08a5f6da-8cc4-4a5a-9666-fe35c9574f95\") " pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" Oct 31 01:09:57.327069 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:57.327029 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:57.419730 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:57.419688 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08a5f6da-8cc4-4a5a-9666-fe35c9574f95-dshm\") pod \"premium-w6-1b8d05-kserve-b68468ddd-xv5fk\" (UID: \"08a5f6da-8cc4-4a5a-9666-fe35c9574f95\") " pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" Oct 31 01:09:57.419916 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:57.419783 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n56wx\" (UniqueName: \"kubernetes.io/projected/08a5f6da-8cc4-4a5a-9666-fe35c9574f95-kube-api-access-n56wx\") pod \"premium-w6-1b8d05-kserve-b68468ddd-xv5fk\" (UID: \"08a5f6da-8cc4-4a5a-9666-fe35c9574f95\") " pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" Oct 31 01:09:57.419916 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:57.419833 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08a5f6da-8cc4-4a5a-9666-fe35c9574f95-home\") pod \"premium-w6-1b8d05-kserve-b68468ddd-xv5fk\" (UID: \"08a5f6da-8cc4-4a5a-9666-fe35c9574f95\") " pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" Oct 31 01:09:57.419916 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:57.419865 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08a5f6da-8cc4-4a5a-9666-fe35c9574f95-model-cache\") pod \"premium-w6-1b8d05-kserve-b68468ddd-xv5fk\" (UID: \"08a5f6da-8cc4-4a5a-9666-fe35c9574f95\") " pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" Oct 31 01:09:57.419916 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:57.419892 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08a5f6da-8cc4-4a5a-9666-fe35c9574f95-tls-certs\") pod \"premium-w6-1b8d05-kserve-b68468ddd-xv5fk\" (UID: \"08a5f6da-8cc4-4a5a-9666-fe35c9574f95\") " pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" Oct 31 01:09:57.419916 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:57.419917 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08a5f6da-8cc4-4a5a-9666-fe35c9574f95-tmp-dir\") pod \"premium-w6-1b8d05-kserve-b68468ddd-xv5fk\" (UID: \"08a5f6da-8cc4-4a5a-9666-fe35c9574f95\") " pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" Oct 31 01:09:57.420267 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:57.420232 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08a5f6da-8cc4-4a5a-9666-fe35c9574f95-model-cache\") pod \"premium-w6-1b8d05-kserve-b68468ddd-xv5fk\" (UID: \"08a5f6da-8cc4-4a5a-9666-fe35c9574f95\") " pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" Oct 31 01:09:57.420386 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:57.420276 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08a5f6da-8cc4-4a5a-9666-fe35c9574f95-tmp-dir\") pod \"premium-w6-1b8d05-kserve-b68468ddd-xv5fk\" (UID: \"08a5f6da-8cc4-4a5a-9666-fe35c9574f95\") " pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" Oct 31 01:09:57.420386 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:57.420309 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08a5f6da-8cc4-4a5a-9666-fe35c9574f95-home\") pod \"premium-w6-1b8d05-kserve-b68468ddd-xv5fk\" (UID: \"08a5f6da-8cc4-4a5a-9666-fe35c9574f95\") " pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" Oct 31 01:09:57.422285 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:57.422253 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08a5f6da-8cc4-4a5a-9666-fe35c9574f95-dshm\") pod \"premium-w6-1b8d05-kserve-b68468ddd-xv5fk\" (UID: \"08a5f6da-8cc4-4a5a-9666-fe35c9574f95\") " pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" Oct 31 01:09:57.422553 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:57.422537 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08a5f6da-8cc4-4a5a-9666-fe35c9574f95-tls-certs\") pod \"premium-w6-1b8d05-kserve-b68468ddd-xv5fk\" (UID: \"08a5f6da-8cc4-4a5a-9666-fe35c9574f95\") " pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" Oct 31 01:09:57.428361 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:57.428317 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n56wx\" (UniqueName: \"kubernetes.io/projected/08a5f6da-8cc4-4a5a-9666-fe35c9574f95-kube-api-access-n56wx\") pod \"premium-w6-1b8d05-kserve-b68468ddd-xv5fk\" (UID: \"08a5f6da-8cc4-4a5a-9666-fe35c9574f95\") " pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" Oct 31 01:09:57.515487 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:57.515368 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" Oct 31 01:09:57.681397 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:57.681354 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:57.794005 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:57.793983 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk"] Oct 31 01:09:57.795529 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:09:57.795496 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08a5f6da_8cc4_4a5a_9666_fe35c9574f95.slice/crio-c1b4a1241c4a7c0c09dca06538bc5cee1e3e00d8f0e2aca3b8ec3ed4a02ca520 WatchSource:0}: Error finding container c1b4a1241c4a7c0c09dca06538bc5cee1e3e00d8f0e2aca3b8ec3ed4a02ca520: Status 404 returned error can't find the container with id c1b4a1241c4a7c0c09dca06538bc5cee1e3e00d8f0e2aca3b8ec3ed4a02ca520 Oct 31 01:09:58.301909 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:58.301867 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" event={"ID":"08a5f6da-8cc4-4a5a-9666-fe35c9574f95","Type":"ContainerStarted","Data":"a490227f3a04ca24e002491fed82910c318ffc6d73a7bb60a390977ae0159cb2"} Oct 31 01:09:58.302329 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:58.301921 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" event={"ID":"08a5f6da-8cc4-4a5a-9666-fe35c9574f95","Type":"ContainerStarted","Data":"c1b4a1241c4a7c0c09dca06538bc5cee1e3e00d8f0e2aca3b8ec3ed4a02ca520"} Oct 31 01:09:58.322337 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:58.322273 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" podStartSLOduration=1.3222568749999999 podStartE2EDuration="1.322256875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:58.319895196 +0000 UTC m=+842.678617514" watchObservedRunningTime="2026-10-31 01:09:58.322256875 +0000 UTC m=+842.680979191" Oct 31 01:09:58.481552 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:58.481512 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:58.515740 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:58.515704 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" Oct 31 01:09:59.353088 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:59.353046 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:59.993090 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:09:59.993056 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:00.689579 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:00.689543 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:01.018326 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:10:01.018223 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:10:01.198330 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:01.198288 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:10:01.375696 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:01.375615 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:01.619379 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:01.619332 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:02.269767 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:02.269732 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:03.000965 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:03.000891 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:03.421469 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:03.421413 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:04.045793 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:04.045756 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:04.701800 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:04.701739 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:05.397196 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:05.397159 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:06.066512 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:06.066476 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:06.792733 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:06.792696 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6"] Oct 31 01:10:06.795214 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:06.795180 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:06.796099 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:06.796078 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" Oct 31 01:10:06.805784 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:06.805755 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-0a6cca-openshift-default-dockercfg-jn6mf\"" Oct 31 01:10:06.840102 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:06.839882 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6"] Oct 31 01:10:06.932576 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:06.932490 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-workload-certs\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6\" (UID: \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" Oct 31 01:10:06.932576 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:06.932529 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-istio-data\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6\" (UID: \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" Oct 31 01:10:06.932802 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:06.932692 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-istio-token\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6\" (UID: \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" Oct 31 01:10:06.932802 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:06.932754 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-istiod-ca-cert\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6\" (UID: \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" Oct 31 01:10:06.932878 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:06.932843 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-istio-podinfo\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6\" (UID: \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" Oct 31 01:10:06.932912 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:06.932897 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-istio-envoy\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6\" (UID: \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" Oct 31 01:10:06.933003 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:06.932980 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t75vm\" (UniqueName: \"kubernetes.io/projected/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-kube-api-access-t75vm\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6\" (UID: \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" Oct 31 01:10:06.933044 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:06.933033 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-credential-socket\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6\" (UID: \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" Oct 31 01:10:06.933125 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:06.933114 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-workload-socket\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6\" (UID: \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" Oct 31 01:10:07.033880 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:07.033829 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t75vm\" (UniqueName: \"kubernetes.io/projected/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-kube-api-access-t75vm\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6\" (UID: \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" Oct 31 01:10:07.034079 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:07.033911 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-credential-socket\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6\" (UID: \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" Oct 31 01:10:07.034079 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:07.033956 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-workload-socket\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6\" (UID: \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" Oct 31 01:10:07.034079 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:07.033997 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-workload-certs\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6\" (UID: \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" Oct 31 01:10:07.034079 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:07.034021 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-istio-data\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6\" (UID: \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" Oct 31 01:10:07.034307 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:07.034118 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-istio-token\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6\" (UID: \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" Oct 31 01:10:07.034307 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:07.034203 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-istiod-ca-cert\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6\" (UID: \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" Oct 31 01:10:07.034307 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:07.034235 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-istio-podinfo\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6\" (UID: \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" Oct 31 01:10:07.034307 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:07.034295 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-istio-envoy\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6\" (UID: \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" Oct 31 01:10:07.034560 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:07.034346 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-credential-socket\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6\" (UID: \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" Oct 31 01:10:07.034560 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:07.034365 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-workload-socket\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6\" (UID: \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" Oct 31 01:10:07.034754 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:07.034731 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-istio-data\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6\" (UID: \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" Oct 31 01:10:07.034822 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:07.034779 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-workload-certs\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6\" (UID: \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" Oct 31 01:10:07.034864 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:07.034828 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-istiod-ca-cert\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6\" (UID: \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" Oct 31 01:10:07.036629 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:07.036608 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-istio-envoy\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6\" (UID: \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" Oct 31 01:10:07.036866 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:07.036844 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-istio-podinfo\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6\" (UID: \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" Oct 31 01:10:07.043312 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:07.043253 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-istio-token\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6\" (UID: \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" Oct 31 01:10:07.043393 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:07.043376 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t75vm\" (UniqueName: \"kubernetes.io/projected/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-kube-api-access-t75vm\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6\" (UID: \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" Oct 31 01:10:07.143637 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:07.143598 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" Oct 31 01:10:07.426913 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:07.426886 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6"] Oct 31 01:10:07.428715 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:10:07.428686 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc34b9f5_85b1_4cab_a49e_c2106d685cc2.slice/crio-9a921b9aa2d87b245ca7de8ec33b82a3d4bd6834c9528b5f525bf79a89938eec WatchSource:0}: Error finding container 9a921b9aa2d87b245ca7de8ec33b82a3d4bd6834c9528b5f525bf79a89938eec: Status 404 returned error can't find the container with id 9a921b9aa2d87b245ca7de8ec33b82a3d4bd6834c9528b5f525bf79a89938eec Oct 31 01:10:07.431596 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:07.431562 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:10:07.431701 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:07.431647 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:10:07.431701 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:07.431692 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:10:07.479953 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:07.479920 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:07.521442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:07.516271 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" Oct 31 01:10:07.535126 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:07.535103 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" Oct 31 01:10:08.195019 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:08.194982 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:08.388102 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:08.388065 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" event={"ID":"dc34b9f5-85b1-4cab-a49e-c2106d685cc2","Type":"ContainerStarted","Data":"107e6b0b6110a5e9993d7259cfe27fa51563593824c65fc1446ac3ea96c881b7"} Oct 31 01:10:08.388289 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:08.388128 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" event={"ID":"dc34b9f5-85b1-4cab-a49e-c2106d685cc2","Type":"ContainerStarted","Data":"9a921b9aa2d87b245ca7de8ec33b82a3d4bd6834c9528b5f525bf79a89938eec"} Oct 31 01:10:08.432441 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:08.427092 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" podStartSLOduration=2.427068793 podStartE2EDuration="2.427068793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:10:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:10:08.410014104 +0000 UTC m=+852.768736433" watchObservedRunningTime="2026-10-31 01:10:08.427068793 +0000 UTC m=+852.785791111" Oct 31 01:10:08.440087 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:08.439891 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" Oct 31 01:10:08.853937 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:08.853882 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:09.144548 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:09.144449 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" Oct 31 01:10:09.146241 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:09.146212 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 31 01:10:09.146397 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:09.146261 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" podUID="dc34b9f5-85b1-4cab-a49e-c2106d685cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 31 01:10:09.486338 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:09.486308 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:10.144154 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:10.144114 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 31 01:10:10.144343 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:10.144173 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" podUID="dc34b9f5-85b1-4cab-a49e-c2106d685cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 31 01:10:10.225438 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:10.225371 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:10.520439 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:10.519638 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:10.646197 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:10.646070 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:11.145444 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:11.144851 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 31 01:10:11.145444 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:11.144916 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" podUID="dc34b9f5-85b1-4cab-a49e-c2106d685cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 31 01:10:11.192951 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:11.192909 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:11.493664 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:11.493458 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:11.970987 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:11.970942 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:12.144162 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:12.144126 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 31 01:10:12.144345 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:12.144187 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" podUID="dc34b9f5-85b1-4cab-a49e-c2106d685cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 31 01:10:12.692821 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:12.692785 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:13.144724 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:13.144630 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 31 01:10:13.144724 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:13.144686 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" podUID="dc34b9f5-85b1-4cab-a49e-c2106d685cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 31 01:10:13.490920 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:13.490880 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:14.142331 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:14.142227 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:14.145039 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:14.144207 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 31 01:10:14.145039 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:14.144261 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" podUID="dc34b9f5-85b1-4cab-a49e-c2106d685cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 31 01:10:14.873390 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:14.873352 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:14.981256 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:14.981208 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:15.148644 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:15.148566 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" Oct 31 01:10:15.149050 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:15.149036 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" Oct 31 01:10:15.149660 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:15.149640 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" Oct 31 01:10:15.696600 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:15.696559 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:15.951691 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:15.951580 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:16.552809 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:16.552764 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:16.721033 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:16.720988 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:10:17.671928 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:17.671892 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:18.340876 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:18.340831 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:18.462702 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:18.462663 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:19.237364 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:19.237333 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:20.091961 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:20.091925 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:20.920286 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:20.920246 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:21.621968 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:21.621928 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:22.326190 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:22.326138 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:23.004891 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:23.004839 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:23.813131 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:23.813072 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:24.491823 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:24.491787 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:25.200344 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:25.200311 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:26.139263 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:26.139216 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:26.948523 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:26.947811 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:28.035439 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:28.034820 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:29.218586 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:29.218547 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:30.305269 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:30.305225 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:31.003803 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:10:31.003766 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:10:31.298746 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:31.298656 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:32.198047 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:32.198003 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:32.912481 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:32.912452 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:33.597121 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:33.597087 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:34.413340 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:34.413298 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:35.094577 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:35.094538 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:35.886590 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:35.886544 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:36.582148 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:36.582112 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:36.630708 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:36.630660 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh"] Oct 31 01:10:36.635501 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:36.635472 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" Oct 31 01:10:36.636214 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:36.636185 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh"] Oct 31 01:10:36.640654 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:36.640622 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-7f6d1e-openshift-default-dockercfg-fjfcz\"" Oct 31 01:10:36.655900 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:36.655869 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-istio-envoy\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh\" (UID: \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" Oct 31 01:10:36.656092 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:36.655912 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-istio-podinfo\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh\" (UID: \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" Oct 31 01:10:36.656092 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:36.655932 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-istiod-ca-cert\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh\" (UID: \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" Oct 31 01:10:36.656214 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:36.656104 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-credential-socket\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh\" (UID: \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" Oct 31 01:10:36.656214 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:36.656147 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-istio-data\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh\" (UID: \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" Oct 31 01:10:36.656314 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:36.656298 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-workload-certs\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh\" (UID: \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" Oct 31 01:10:36.656489 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:36.656468 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-istio-token\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh\" (UID: \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" Oct 31 01:10:36.656556 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:36.656534 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddwsg\" (UniqueName: \"kubernetes.io/projected/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-kube-api-access-ddwsg\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh\" (UID: \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" Oct 31 01:10:36.656694 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:36.656611 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-workload-socket\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh\" (UID: \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" Oct 31 01:10:36.757388 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:36.757338 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-istio-token\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh\" (UID: \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" Oct 31 01:10:36.757627 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:36.757442 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddwsg\" (UniqueName: \"kubernetes.io/projected/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-kube-api-access-ddwsg\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh\" (UID: \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" Oct 31 01:10:36.757627 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:36.757479 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-workload-socket\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh\" (UID: \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" Oct 31 01:10:36.757627 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:36.757539 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-istio-envoy\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh\" (UID: \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" Oct 31 01:10:36.757627 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:36.757581 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-istio-podinfo\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh\" (UID: \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" Oct 31 01:10:36.757627 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:36.757599 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-istiod-ca-cert\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh\" (UID: \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" Oct 31 01:10:36.757930 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:36.757640 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-credential-socket\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh\" (UID: \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" Oct 31 01:10:36.757930 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:36.757666 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-istio-data\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh\" (UID: \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" Oct 31 01:10:36.757930 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:36.757764 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-workload-certs\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh\" (UID: \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" Oct 31 01:10:36.758095 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:36.757930 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-workload-socket\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh\" (UID: \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" Oct 31 01:10:36.758095 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:36.758054 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-credential-socket\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh\" (UID: \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" Oct 31 01:10:36.758247 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:36.758230 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-workload-certs\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh\" (UID: \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" Oct 31 01:10:36.758344 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:36.758319 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-istio-data\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh\" (UID: \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" Oct 31 01:10:36.758626 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:36.758601 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-istiod-ca-cert\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh\" (UID: \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" Oct 31 01:10:36.760274 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:36.760251 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-istio-envoy\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh\" (UID: \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" Oct 31 01:10:36.761159 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:36.761130 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-istio-podinfo\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh\" (UID: \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" Oct 31 01:10:36.766160 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:36.766128 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-istio-token\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh\" (UID: \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" Oct 31 01:10:36.766668 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:36.766645 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddwsg\" (UniqueName: \"kubernetes.io/projected/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-kube-api-access-ddwsg\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh\" (UID: \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" Oct 31 01:10:36.954848 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:36.954788 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" Oct 31 01:10:37.113641 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.113606 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8"] Oct 31 01:10:37.117433 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.117394 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" Oct 31 01:10:37.125185 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.124035 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-28ff08-openshift-default-dockercfg-gpfzz\"" Oct 31 01:10:37.131310 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.131280 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8"] Oct 31 01:10:37.163017 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.162966 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81095022-acb2-4b1c-acc6-a9501b0ea478-istio-podinfo\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8\" (UID: \"81095022-acb2-4b1c-acc6-a9501b0ea478\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" Oct 31 01:10:37.163017 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.163028 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81095022-acb2-4b1c-acc6-a9501b0ea478-credential-socket\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8\" (UID: \"81095022-acb2-4b1c-acc6-a9501b0ea478\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" Oct 31 01:10:37.163284 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.163135 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81095022-acb2-4b1c-acc6-a9501b0ea478-istio-envoy\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8\" (UID: \"81095022-acb2-4b1c-acc6-a9501b0ea478\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" Oct 31 01:10:37.163284 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.163249 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66kb5\" (UniqueName: \"kubernetes.io/projected/81095022-acb2-4b1c-acc6-a9501b0ea478-kube-api-access-66kb5\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8\" (UID: \"81095022-acb2-4b1c-acc6-a9501b0ea478\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" Oct 31 01:10:37.163432 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.163298 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81095022-acb2-4b1c-acc6-a9501b0ea478-workload-certs\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8\" (UID: \"81095022-acb2-4b1c-acc6-a9501b0ea478\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" Oct 31 01:10:37.163432 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.163395 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81095022-acb2-4b1c-acc6-a9501b0ea478-istio-data\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8\" (UID: \"81095022-acb2-4b1c-acc6-a9501b0ea478\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" Oct 31 01:10:37.163552 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.163469 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81095022-acb2-4b1c-acc6-a9501b0ea478-istio-token\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8\" (UID: \"81095022-acb2-4b1c-acc6-a9501b0ea478\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" Oct 31 01:10:37.163552 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.163507 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81095022-acb2-4b1c-acc6-a9501b0ea478-workload-socket\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8\" (UID: \"81095022-acb2-4b1c-acc6-a9501b0ea478\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" Oct 31 01:10:37.163634 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.163608 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81095022-acb2-4b1c-acc6-a9501b0ea478-istiod-ca-cert\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8\" (UID: \"81095022-acb2-4b1c-acc6-a9501b0ea478\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" Oct 31 01:10:37.245880 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:10:37.245849 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f8c2432_9405_4f4f_b24f_1b73ea0da86a.slice/crio-0efd785b943e9ba0cb070853bb337f6f6df533d39460ee9055512b1e7832b307 WatchSource:0}: Error finding container 0efd785b943e9ba0cb070853bb337f6f6df533d39460ee9055512b1e7832b307: Status 404 returned error can't find the container with id 0efd785b943e9ba0cb070853bb337f6f6df533d39460ee9055512b1e7832b307 Oct 31 01:10:37.246189 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.246163 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh"] Oct 31 01:10:37.248687 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.248656 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:10:37.248810 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.248731 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:10:37.248810 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.248777 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:10:37.265373 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.265342 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81095022-acb2-4b1c-acc6-a9501b0ea478-istio-podinfo\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8\" (UID: \"81095022-acb2-4b1c-acc6-a9501b0ea478\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" Oct 31 01:10:37.265459 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.265396 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81095022-acb2-4b1c-acc6-a9501b0ea478-credential-socket\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8\" (UID: \"81095022-acb2-4b1c-acc6-a9501b0ea478\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" Oct 31 01:10:37.265510 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.265462 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81095022-acb2-4b1c-acc6-a9501b0ea478-istio-envoy\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8\" (UID: \"81095022-acb2-4b1c-acc6-a9501b0ea478\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" Oct 31 01:10:37.265558 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.265507 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66kb5\" (UniqueName: \"kubernetes.io/projected/81095022-acb2-4b1c-acc6-a9501b0ea478-kube-api-access-66kb5\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8\" (UID: \"81095022-acb2-4b1c-acc6-a9501b0ea478\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" Oct 31 01:10:37.265558 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.265535 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81095022-acb2-4b1c-acc6-a9501b0ea478-workload-certs\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8\" (UID: \"81095022-acb2-4b1c-acc6-a9501b0ea478\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" Oct 31 01:10:37.265656 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.265567 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81095022-acb2-4b1c-acc6-a9501b0ea478-istio-data\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8\" (UID: \"81095022-acb2-4b1c-acc6-a9501b0ea478\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" Oct 31 01:10:37.265656 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.265609 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81095022-acb2-4b1c-acc6-a9501b0ea478-istio-token\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8\" (UID: \"81095022-acb2-4b1c-acc6-a9501b0ea478\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" Oct 31 01:10:37.265656 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.265644 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81095022-acb2-4b1c-acc6-a9501b0ea478-workload-socket\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8\" (UID: \"81095022-acb2-4b1c-acc6-a9501b0ea478\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" Oct 31 01:10:37.265802 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.265693 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81095022-acb2-4b1c-acc6-a9501b0ea478-istiod-ca-cert\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8\" (UID: \"81095022-acb2-4b1c-acc6-a9501b0ea478\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" Oct 31 01:10:37.265897 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.265857 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81095022-acb2-4b1c-acc6-a9501b0ea478-credential-socket\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8\" (UID: \"81095022-acb2-4b1c-acc6-a9501b0ea478\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" Oct 31 01:10:37.265897 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.265880 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81095022-acb2-4b1c-acc6-a9501b0ea478-workload-certs\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8\" (UID: \"81095022-acb2-4b1c-acc6-a9501b0ea478\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" Oct 31 01:10:37.266090 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.266059 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81095022-acb2-4b1c-acc6-a9501b0ea478-istio-data\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8\" (UID: \"81095022-acb2-4b1c-acc6-a9501b0ea478\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" Oct 31 01:10:37.266168 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.266159 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81095022-acb2-4b1c-acc6-a9501b0ea478-workload-socket\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8\" (UID: \"81095022-acb2-4b1c-acc6-a9501b0ea478\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" Oct 31 01:10:37.266518 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.266495 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81095022-acb2-4b1c-acc6-a9501b0ea478-istiod-ca-cert\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8\" (UID: \"81095022-acb2-4b1c-acc6-a9501b0ea478\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" Oct 31 01:10:37.267954 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.267926 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81095022-acb2-4b1c-acc6-a9501b0ea478-istio-envoy\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8\" (UID: \"81095022-acb2-4b1c-acc6-a9501b0ea478\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" Oct 31 01:10:37.268243 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.268193 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81095022-acb2-4b1c-acc6-a9501b0ea478-istio-podinfo\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8\" (UID: \"81095022-acb2-4b1c-acc6-a9501b0ea478\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" Oct 31 01:10:37.276159 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.276135 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81095022-acb2-4b1c-acc6-a9501b0ea478-istio-token\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8\" (UID: \"81095022-acb2-4b1c-acc6-a9501b0ea478\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" Oct 31 01:10:37.276251 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.276226 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66kb5\" (UniqueName: \"kubernetes.io/projected/81095022-acb2-4b1c-acc6-a9501b0ea478-kube-api-access-66kb5\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8\" (UID: \"81095022-acb2-4b1c-acc6-a9501b0ea478\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" Oct 31 01:10:37.437955 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.435665 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" Oct 31 01:10:37.463428 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.463380 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:10:37.472548 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.472487 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:37.694761 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.694725 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" event={"ID":"8f8c2432-9405-4f4f-b24f-1b73ea0da86a","Type":"ContainerStarted","Data":"3268df394c99ef1132a6a6b2280a81753857a6722c453d7681cb7f6e93d56da6"} Oct 31 01:10:37.695197 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.694785 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" event={"ID":"8f8c2432-9405-4f4f-b24f-1b73ea0da86a","Type":"ContainerStarted","Data":"0efd785b943e9ba0cb070853bb337f6f6df533d39460ee9055512b1e7832b307"} Oct 31 01:10:37.723801 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.723744 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" podStartSLOduration=1.723726131 podStartE2EDuration="1.723726131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:10:37.720163633 +0000 UTC m=+882.078885951" watchObservedRunningTime="2026-10-31 01:10:37.723726131 +0000 UTC m=+882.082448447" Oct 31 01:10:37.764503 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.761681 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8"] Oct 31 01:10:37.769899 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.769844 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:10:37.770260 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.769995 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:10:37.770260 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.770087 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:10:37.957446 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:37.956602 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" Oct 31 01:10:38.212649 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:38.212597 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:38.724443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:38.724167 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" event={"ID":"81095022-acb2-4b1c-acc6-a9501b0ea478","Type":"ContainerStarted","Data":"eade63983f78f236269934cc635215faffbfcadf3990e2d1fb7b34ccf28dda43"} Oct 31 01:10:38.724443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:38.724225 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" event={"ID":"81095022-acb2-4b1c-acc6-a9501b0ea478","Type":"ContainerStarted","Data":"18ac5f401f72290cf5f2c8fc8b1690dfee260038e49a7b228181d2202fef794e"} Oct 31 01:10:38.791443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:38.790665 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" podStartSLOduration=1.790644362 podStartE2EDuration="1.790644362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:10:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:10:38.766049726 +0000 UTC m=+883.124772042" watchObservedRunningTime="2026-10-31 01:10:38.790644362 +0000 UTC m=+883.149366680" Oct 31 01:10:38.957705 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:38.957660 4440 patch_prober.go:29] interesting pod/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 01:10:38.958108 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:38.957969 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" podUID="8f8c2432-9405-4f4f-b24f-1b73ea0da86a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 01:10:39.005341 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:39.005235 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:39.442445 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:39.441759 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" Oct 31 01:10:39.446064 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:39.446026 4440 patch_prober.go:29] interesting pod/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 01:10:39.446318 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:39.446288 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" podUID="81095022-acb2-4b1c-acc6-a9501b0ea478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 01:10:39.819474 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:39.819344 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:39.955961 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:39.955923 4440 patch_prober.go:29] interesting pod/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 01:10:39.956148 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:39.955976 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" podUID="8f8c2432-9405-4f4f-b24f-1b73ea0da86a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 01:10:40.439660 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:40.439622 4440 patch_prober.go:29] interesting pod/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 01:10:40.439965 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:40.439933 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" podUID="81095022-acb2-4b1c-acc6-a9501b0ea478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 01:10:40.820509 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:40.817995 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:40.955246 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:40.955217 4440 patch_prober.go:29] interesting pod/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 01:10:40.955521 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:40.955497 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" podUID="8f8c2432-9405-4f4f-b24f-1b73ea0da86a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 01:10:41.451918 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:41.451886 4440 patch_prober.go:29] interesting pod/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 01:10:41.452181 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:41.452155 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" podUID="81095022-acb2-4b1c-acc6-a9501b0ea478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 01:10:41.624828 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:41.618119 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:41.793126 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:41.793016 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:41.956045 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:41.956001 4440 patch_prober.go:29] interesting pod/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 01:10:41.956663 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:41.956618 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" podUID="8f8c2432-9405-4f4f-b24f-1b73ea0da86a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 01:10:42.360231 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:42.360174 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:42.437443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:42.436610 4440 patch_prober.go:29] interesting pod/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 01:10:42.437443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:42.436668 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" podUID="81095022-acb2-4b1c-acc6-a9501b0ea478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 01:10:42.657298 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:42.657186 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:42.956071 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:42.956029 4440 patch_prober.go:29] interesting pod/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 01:10:42.956535 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:42.956498 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" podUID="8f8c2432-9405-4f4f-b24f-1b73ea0da86a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 01:10:43.186854 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:43.186819 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:43.438447 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:43.436914 4440 patch_prober.go:29] interesting pod/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 01:10:43.438447 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:43.436976 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" podUID="81095022-acb2-4b1c-acc6-a9501b0ea478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 01:10:43.955170 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:43.955130 4440 patch_prober.go:29] interesting pod/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 01:10:43.955374 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:43.955199 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" podUID="8f8c2432-9405-4f4f-b24f-1b73ea0da86a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 01:10:44.096533 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:44.096500 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:44.436999 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:44.436961 4440 patch_prober.go:29] interesting pod/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 01:10:44.437212 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:44.437032 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" podUID="81095022-acb2-4b1c-acc6-a9501b0ea478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 01:10:45.056934 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:45.056902 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:45.180094 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:45.180058 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" Oct 31 01:10:45.180616 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:45.180512 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" Oct 31 01:10:45.181331 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:45.181314 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" Oct 31 01:10:45.440918 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:45.440886 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" Oct 31 01:10:45.441866 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:45.441524 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" Oct 31 01:10:45.442804 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:45.442781 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" Oct 31 01:10:45.741839 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:45.741739 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:45.849807 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:45.849761 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:46.481809 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:46.481772 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:47.215489 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:47.215455 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:47.976838 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:47.976803 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:48.600094 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:48.600056 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:49.202712 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:49.202675 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:49.861077 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:49.861034 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:50.504895 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:50.504844 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:51.173502 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:51.173468 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:51.790189 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:51.790151 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:52.000966 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:52.000929 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv"] Oct 31 01:10:52.006007 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:52.005984 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" Oct 31 01:10:52.022199 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:52.021901 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv"] Oct 31 01:10:52.054692 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:52.054625 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-8499c8-openshift-default-dockercfg-h4hkj\"" Oct 31 01:10:52.140661 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:52.140610 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c523fbe-6cf8-456c-acc0-463ff721c1da-workload-socket\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv\" (UID: \"1c523fbe-6cf8-456c-acc0-463ff721c1da\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" Oct 31 01:10:52.140888 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:52.140677 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c523fbe-6cf8-456c-acc0-463ff721c1da-istio-data\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv\" (UID: \"1c523fbe-6cf8-456c-acc0-463ff721c1da\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" Oct 31 01:10:52.140888 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:52.140769 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c523fbe-6cf8-456c-acc0-463ff721c1da-credential-socket\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv\" (UID: \"1c523fbe-6cf8-456c-acc0-463ff721c1da\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" Oct 31 01:10:52.140888 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:52.140793 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c523fbe-6cf8-456c-acc0-463ff721c1da-istio-envoy\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv\" (UID: \"1c523fbe-6cf8-456c-acc0-463ff721c1da\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" Oct 31 01:10:52.140888 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:52.140822 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c523fbe-6cf8-456c-acc0-463ff721c1da-istio-token\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv\" (UID: \"1c523fbe-6cf8-456c-acc0-463ff721c1da\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" Oct 31 01:10:52.141113 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:52.140939 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c523fbe-6cf8-456c-acc0-463ff721c1da-istiod-ca-cert\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv\" (UID: \"1c523fbe-6cf8-456c-acc0-463ff721c1da\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" Oct 31 01:10:52.141113 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:52.140974 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvvl6\" (UniqueName: \"kubernetes.io/projected/1c523fbe-6cf8-456c-acc0-463ff721c1da-kube-api-access-mvvl6\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv\" (UID: \"1c523fbe-6cf8-456c-acc0-463ff721c1da\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" Oct 31 01:10:52.141218 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:52.141139 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c523fbe-6cf8-456c-acc0-463ff721c1da-istio-podinfo\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv\" (UID: \"1c523fbe-6cf8-456c-acc0-463ff721c1da\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" Oct 31 01:10:52.141282 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:52.141248 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c523fbe-6cf8-456c-acc0-463ff721c1da-workload-certs\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv\" (UID: \"1c523fbe-6cf8-456c-acc0-463ff721c1da\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" Oct 31 01:10:52.242723 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:52.242684 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c523fbe-6cf8-456c-acc0-463ff721c1da-workload-socket\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv\" (UID: \"1c523fbe-6cf8-456c-acc0-463ff721c1da\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" Oct 31 01:10:52.242930 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:52.242735 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c523fbe-6cf8-456c-acc0-463ff721c1da-istio-data\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv\" (UID: \"1c523fbe-6cf8-456c-acc0-463ff721c1da\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" Oct 31 01:10:52.242930 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:52.242758 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c523fbe-6cf8-456c-acc0-463ff721c1da-credential-socket\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv\" (UID: \"1c523fbe-6cf8-456c-acc0-463ff721c1da\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" Oct 31 01:10:52.242930 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:52.242876 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c523fbe-6cf8-456c-acc0-463ff721c1da-istio-envoy\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv\" (UID: \"1c523fbe-6cf8-456c-acc0-463ff721c1da\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" Oct 31 01:10:52.242930 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:52.242929 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c523fbe-6cf8-456c-acc0-463ff721c1da-istio-token\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv\" (UID: \"1c523fbe-6cf8-456c-acc0-463ff721c1da\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" Oct 31 01:10:52.243160 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:52.242972 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c523fbe-6cf8-456c-acc0-463ff721c1da-istiod-ca-cert\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv\" (UID: \"1c523fbe-6cf8-456c-acc0-463ff721c1da\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" Oct 31 01:10:52.243160 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:52.243004 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvvl6\" (UniqueName: \"kubernetes.io/projected/1c523fbe-6cf8-456c-acc0-463ff721c1da-kube-api-access-mvvl6\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv\" (UID: \"1c523fbe-6cf8-456c-acc0-463ff721c1da\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" Oct 31 01:10:52.243160 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:52.243131 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c523fbe-6cf8-456c-acc0-463ff721c1da-istio-podinfo\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv\" (UID: \"1c523fbe-6cf8-456c-acc0-463ff721c1da\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" Oct 31 01:10:52.243311 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:52.243172 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c523fbe-6cf8-456c-acc0-463ff721c1da-workload-socket\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv\" (UID: \"1c523fbe-6cf8-456c-acc0-463ff721c1da\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" Oct 31 01:10:52.243311 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:52.243206 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c523fbe-6cf8-456c-acc0-463ff721c1da-workload-certs\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv\" (UID: \"1c523fbe-6cf8-456c-acc0-463ff721c1da\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" Oct 31 01:10:52.243750 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:52.243725 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c523fbe-6cf8-456c-acc0-463ff721c1da-istio-data\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv\" (UID: \"1c523fbe-6cf8-456c-acc0-463ff721c1da\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" Oct 31 01:10:52.243750 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:52.243737 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c523fbe-6cf8-456c-acc0-463ff721c1da-credential-socket\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv\" (UID: \"1c523fbe-6cf8-456c-acc0-463ff721c1da\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" Oct 31 01:10:52.243899 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:52.243763 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c523fbe-6cf8-456c-acc0-463ff721c1da-workload-certs\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv\" (UID: \"1c523fbe-6cf8-456c-acc0-463ff721c1da\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" Oct 31 01:10:52.244150 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:52.244129 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c523fbe-6cf8-456c-acc0-463ff721c1da-istiod-ca-cert\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv\" (UID: \"1c523fbe-6cf8-456c-acc0-463ff721c1da\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" Oct 31 01:10:52.245561 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:52.245537 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c523fbe-6cf8-456c-acc0-463ff721c1da-istio-podinfo\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv\" (UID: \"1c523fbe-6cf8-456c-acc0-463ff721c1da\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" Oct 31 01:10:52.246785 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:52.246759 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c523fbe-6cf8-456c-acc0-463ff721c1da-istio-envoy\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv\" (UID: \"1c523fbe-6cf8-456c-acc0-463ff721c1da\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" Oct 31 01:10:52.251279 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:52.251248 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c523fbe-6cf8-456c-acc0-463ff721c1da-istio-token\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv\" (UID: \"1c523fbe-6cf8-456c-acc0-463ff721c1da\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" Oct 31 01:10:52.251387 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:52.251277 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvvl6\" (UniqueName: \"kubernetes.io/projected/1c523fbe-6cf8-456c-acc0-463ff721c1da-kube-api-access-mvvl6\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv\" (UID: \"1c523fbe-6cf8-456c-acc0-463ff721c1da\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" Oct 31 01:10:52.344660 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:52.344556 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" Oct 31 01:10:52.590736 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:52.590693 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:52.664862 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:52.664829 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv"] Oct 31 01:10:52.665661 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:10:52.665630 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c523fbe_6cf8_456c_acc0_463ff721c1da.slice/crio-7dd11996c63e3dc5ea1e1036d5209b834bd298c5c5129c12b6c36e2d96b40d02 WatchSource:0}: Error finding container 7dd11996c63e3dc5ea1e1036d5209b834bd298c5c5129c12b6c36e2d96b40d02: Status 404 returned error can't find the container with id 7dd11996c63e3dc5ea1e1036d5209b834bd298c5c5129c12b6c36e2d96b40d02 Oct 31 01:10:52.668148 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:52.668117 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:10:52.668228 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:52.668182 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:10:52.668228 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:52.668213 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:10:52.853090 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:52.853051 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" event={"ID":"1c523fbe-6cf8-456c-acc0-463ff721c1da","Type":"ContainerStarted","Data":"a935fc080eda1214bfe404dd4704bcde9e7ac57e22580a6caef58d53be165377"} Oct 31 01:10:52.853473 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:52.853113 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" event={"ID":"1c523fbe-6cf8-456c-acc0-463ff721c1da","Type":"ContainerStarted","Data":"7dd11996c63e3dc5ea1e1036d5209b834bd298c5c5129c12b6c36e2d96b40d02"} Oct 31 01:10:52.876086 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:52.876037 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" podStartSLOduration=1.8760241180000001 podStartE2EDuration="1.876024118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:10:52.873801262 +0000 UTC m=+897.232523577" watchObservedRunningTime="2026-10-31 01:10:52.876024118 +0000 UTC m=+897.234746440" Oct 31 01:10:53.345072 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:53.345032 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" Oct 31 01:10:53.346403 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:53.346375 4440 patch_prober.go:29] interesting pod/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 01:10:53.346558 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:53.346450 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" podUID="1c523fbe-6cf8-456c-acc0-463ff721c1da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 01:10:53.485029 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:53.484993 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:54.342362 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:54.342325 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:54.345531 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:54.345502 4440 patch_prober.go:29] interesting pod/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 01:10:54.345678 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:54.345553 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" podUID="1c523fbe-6cf8-456c-acc0-463ff721c1da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 01:10:55.115441 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:55.114921 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:55.346096 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:55.346057 4440 patch_prober.go:29] interesting pod/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 01:10:55.346522 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:55.346125 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" podUID="1c523fbe-6cf8-456c-acc0-463ff721c1da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 01:10:56.097465 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:56.095905 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:56.345971 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:56.345932 4440 patch_prober.go:29] interesting pod/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 01:10:56.346153 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:56.346000 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" podUID="1c523fbe-6cf8-456c-acc0-463ff721c1da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 01:10:57.065599 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:57.065557 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:57.345054 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:57.344972 4440 patch_prober.go:29] interesting pod/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 01:10:57.345054 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:57.345029 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" podUID="1c523fbe-6cf8-456c-acc0-463ff721c1da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 01:10:57.890270 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:57.890231 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:58.345992 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:58.345954 4440 patch_prober.go:29] interesting pod/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 01:10:58.346168 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:58.346024 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" podUID="1c523fbe-6cf8-456c-acc0-463ff721c1da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 01:10:58.761236 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:58.761196 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:59.483609 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:59.483578 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" Oct 31 01:10:59.484061 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:59.484039 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" Oct 31 01:10:59.484938 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:59.484912 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" Oct 31 01:10:59.712281 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:10:59.712244 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:00.488663 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:00.488628 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:01.000803 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:11:01.000761 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:11:01.118929 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:01.118886 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:01.793589 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:01.793551 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:02.539040 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:02.538997 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:03.206641 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:03.206605 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:03.896351 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:03.896306 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:04.883711 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:04.883669 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:05.565819 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:05.565786 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:06.175618 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:06.175585 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:06.786856 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:06.786816 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:07.697265 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:07.697228 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:08.589091 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:08.589053 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:09.285596 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:09.285557 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:09.991183 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:09.991144 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:10.601152 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:10.601112 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:11.294799 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:11.294763 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:12.086595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:12.086552 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:12.882409 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:12.882376 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:13.683082 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:13.683038 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:14.491693 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:14.491654 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:15.163266 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:15.163233 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:15.784318 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:15.784277 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:16.462928 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:16.462881 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:17.196403 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:17.193694 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:17.994360 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:17.994322 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:18.732763 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:18.732723 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:19.486899 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:19.486863 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:20.187466 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:20.187410 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:20.810851 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:20.810817 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:21.591557 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:21.591519 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:22.457840 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:22.457803 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:23.564614 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:23.564579 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:24.520690 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:24.520642 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:24.893384 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:24.893282 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw"] Oct 31 01:11:24.898624 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:24.898598 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" Oct 31 01:11:24.903479 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:24.903452 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw"] Oct 31 01:11:24.930411 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:24.930384 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-1b8d05\"/\"distinct-w6-1b8d05-kserve-self-signed-certs\"" Oct 31 01:11:24.983283 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:24.983251 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a953e39-91e4-42cc-b7b4-00abdda9e139-tls-certs\") pod \"distinct-w6-1b8d05-kserve-77976c577f-qn6zw\" (UID: \"1a953e39-91e4-42cc-b7b4-00abdda9e139\") " pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" Oct 31 01:11:24.983283 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:24.983292 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a953e39-91e4-42cc-b7b4-00abdda9e139-model-cache\") pod \"distinct-w6-1b8d05-kserve-77976c577f-qn6zw\" (UID: \"1a953e39-91e4-42cc-b7b4-00abdda9e139\") " pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" Oct 31 01:11:24.983550 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:24.983376 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6ngf\" (UniqueName: \"kubernetes.io/projected/1a953e39-91e4-42cc-b7b4-00abdda9e139-kube-api-access-v6ngf\") pod \"distinct-w6-1b8d05-kserve-77976c577f-qn6zw\" (UID: \"1a953e39-91e4-42cc-b7b4-00abdda9e139\") " pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" Oct 31 01:11:24.983550 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:24.983406 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a953e39-91e4-42cc-b7b4-00abdda9e139-home\") pod \"distinct-w6-1b8d05-kserve-77976c577f-qn6zw\" (UID: \"1a953e39-91e4-42cc-b7b4-00abdda9e139\") " pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" Oct 31 01:11:24.983550 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:24.983447 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a953e39-91e4-42cc-b7b4-00abdda9e139-tmp-dir\") pod \"distinct-w6-1b8d05-kserve-77976c577f-qn6zw\" (UID: \"1a953e39-91e4-42cc-b7b4-00abdda9e139\") " pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" Oct 31 01:11:24.983703 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:24.983568 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a953e39-91e4-42cc-b7b4-00abdda9e139-dshm\") pod \"distinct-w6-1b8d05-kserve-77976c577f-qn6zw\" (UID: \"1a953e39-91e4-42cc-b7b4-00abdda9e139\") " pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" Oct 31 01:11:24.994585 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:24.994558 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:25.084697 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:25.084658 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a953e39-91e4-42cc-b7b4-00abdda9e139-tls-certs\") pod \"distinct-w6-1b8d05-kserve-77976c577f-qn6zw\" (UID: \"1a953e39-91e4-42cc-b7b4-00abdda9e139\") " pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" Oct 31 01:11:25.084950 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:25.084718 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a953e39-91e4-42cc-b7b4-00abdda9e139-model-cache\") pod \"distinct-w6-1b8d05-kserve-77976c577f-qn6zw\" (UID: \"1a953e39-91e4-42cc-b7b4-00abdda9e139\") " pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" Oct 31 01:11:25.084950 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:25.084777 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6ngf\" (UniqueName: \"kubernetes.io/projected/1a953e39-91e4-42cc-b7b4-00abdda9e139-kube-api-access-v6ngf\") pod \"distinct-w6-1b8d05-kserve-77976c577f-qn6zw\" (UID: \"1a953e39-91e4-42cc-b7b4-00abdda9e139\") " pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" Oct 31 01:11:25.084950 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:25.084811 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a953e39-91e4-42cc-b7b4-00abdda9e139-home\") pod \"distinct-w6-1b8d05-kserve-77976c577f-qn6zw\" (UID: \"1a953e39-91e4-42cc-b7b4-00abdda9e139\") " pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" Oct 31 01:11:25.084950 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:25.084849 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a953e39-91e4-42cc-b7b4-00abdda9e139-tmp-dir\") pod \"distinct-w6-1b8d05-kserve-77976c577f-qn6zw\" (UID: \"1a953e39-91e4-42cc-b7b4-00abdda9e139\") " pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" Oct 31 01:11:25.084950 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:25.084891 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a953e39-91e4-42cc-b7b4-00abdda9e139-dshm\") pod \"distinct-w6-1b8d05-kserve-77976c577f-qn6zw\" (UID: \"1a953e39-91e4-42cc-b7b4-00abdda9e139\") " pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" Oct 31 01:11:25.085802 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:25.085776 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a953e39-91e4-42cc-b7b4-00abdda9e139-model-cache\") pod \"distinct-w6-1b8d05-kserve-77976c577f-qn6zw\" (UID: \"1a953e39-91e4-42cc-b7b4-00abdda9e139\") " pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" Oct 31 01:11:25.086042 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:25.086020 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a953e39-91e4-42cc-b7b4-00abdda9e139-home\") pod \"distinct-w6-1b8d05-kserve-77976c577f-qn6zw\" (UID: \"1a953e39-91e4-42cc-b7b4-00abdda9e139\") " pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" Oct 31 01:11:25.086548 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:25.086296 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a953e39-91e4-42cc-b7b4-00abdda9e139-tmp-dir\") pod \"distinct-w6-1b8d05-kserve-77976c577f-qn6zw\" (UID: \"1a953e39-91e4-42cc-b7b4-00abdda9e139\") " pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" Oct 31 01:11:25.087995 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:25.087968 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a953e39-91e4-42cc-b7b4-00abdda9e139-tls-certs\") pod \"distinct-w6-1b8d05-kserve-77976c577f-qn6zw\" (UID: \"1a953e39-91e4-42cc-b7b4-00abdda9e139\") " pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" Oct 31 01:11:25.093873 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:25.093852 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6ngf\" (UniqueName: \"kubernetes.io/projected/1a953e39-91e4-42cc-b7b4-00abdda9e139-kube-api-access-v6ngf\") pod \"distinct-w6-1b8d05-kserve-77976c577f-qn6zw\" (UID: \"1a953e39-91e4-42cc-b7b4-00abdda9e139\") " pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" Oct 31 01:11:25.094187 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:25.094164 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a953e39-91e4-42cc-b7b4-00abdda9e139-dshm\") pod \"distinct-w6-1b8d05-kserve-77976c577f-qn6zw\" (UID: \"1a953e39-91e4-42cc-b7b4-00abdda9e139\") " pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" Oct 31 01:11:25.217918 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:25.217888 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" Oct 31 01:11:25.508300 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:25.508267 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw"] Oct 31 01:11:25.509126 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:11:25.509094 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a953e39_91e4_42cc_b7b4_00abdda9e139.slice/crio-0e3bbccb5d8ff85569842ee9469a1164c8224f75039e6cc7f829569918bee5c1 WatchSource:0}: Error finding container 0e3bbccb5d8ff85569842ee9469a1164c8224f75039e6cc7f829569918bee5c1: Status 404 returned error can't find the container with id 0e3bbccb5d8ff85569842ee9469a1164c8224f75039e6cc7f829569918bee5c1 Oct 31 01:11:25.654445 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:25.651007 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:26.146317 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:26.146285 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" event={"ID":"1a953e39-91e4-42cc-b7b4-00abdda9e139","Type":"ContainerStarted","Data":"ae7ac08bab1f3b15d991bdbc0242d420e6d7e781cd4833f6dad4f2723c4e13dd"} Oct 31 01:11:26.146721 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:26.146325 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" event={"ID":"1a953e39-91e4-42cc-b7b4-00abdda9e139","Type":"ContainerStarted","Data":"0e3bbccb5d8ff85569842ee9469a1164c8224f75039e6cc7f829569918bee5c1"} Oct 31 01:11:26.162295 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:26.162240 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" podStartSLOduration=2.162223905 podStartE2EDuration="2.162223905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:11:26.161612571 +0000 UTC m=+930.520334888" watchObservedRunningTime="2026-10-31 01:11:26.162223905 +0000 UTC m=+930.520946222" Oct 31 01:11:26.218654 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:26.218625 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" Oct 31 01:11:26.432973 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:26.432937 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:27.183630 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:27.183595 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:27.893068 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:27.893032 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:28.587462 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:28.587396 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:29.361281 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:29.361236 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:29.682817 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:29.682780 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:11:30.106978 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:30.106877 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:31.237904 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:11:31.237864 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:11:31.816623 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:31.816528 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:32.800188 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:32.800151 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:34.057442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:34.022561 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:34.516562 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:34.516518 4440 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6rjmk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:11:34.516872 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:34.516838 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" podUID="dfa83ced-0142-4d9a-9706-7edb28fda974" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:11:34.850533 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:34.843120 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:35.244538 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:35.244493 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" Oct 31 01:11:35.309613 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:35.309575 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" Oct 31 01:11:35.545998 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:35.545845 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" Oct 31 01:11:35.724799 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:35.721152 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:36.677281 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:36.677246 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:37.812900 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:37.812739 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:38.871799 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:38.871754 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:39.936257 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:39.936209 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg"] Oct 31 01:11:39.942627 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:39.942600 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" Oct 31 01:11:39.949070 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:39.949038 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg"] Oct 31 01:11:39.954908 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:39.954880 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-487a6c\"/\"openshift-service-ca.crt\"" Oct 31 01:11:39.955056 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:39.955035 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-487a6c\"/\"distinct-w3-487a6c-kserve-self-signed-certs\"" Oct 31 01:11:39.955170 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:39.955052 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-487a6c\"/\"kube-root-ca.crt\"" Oct 31 01:11:39.955758 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:39.955740 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-487a6c\"/\"default-dockercfg-9hlbr\"" Oct 31 01:11:40.021856 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:40.021824 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:40.032346 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:40.032320 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/054f9e47-675f-4722-b743-79bbdc826585-dshm\") pod \"distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg\" (UID: \"054f9e47-675f-4722-b743-79bbdc826585\") " pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" Oct 31 01:11:40.032488 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:40.032452 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/054f9e47-675f-4722-b743-79bbdc826585-tls-certs\") pod \"distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg\" (UID: \"054f9e47-675f-4722-b743-79bbdc826585\") " pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" Oct 31 01:11:40.032488 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:40.032479 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/054f9e47-675f-4722-b743-79bbdc826585-model-cache\") pod \"distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg\" (UID: \"054f9e47-675f-4722-b743-79bbdc826585\") " pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" Oct 31 01:11:40.032598 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:40.032535 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/054f9e47-675f-4722-b743-79bbdc826585-home\") pod \"distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg\" (UID: \"054f9e47-675f-4722-b743-79bbdc826585\") " pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" Oct 31 01:11:40.032647 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:40.032589 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/054f9e47-675f-4722-b743-79bbdc826585-tmp-dir\") pod \"distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg\" (UID: \"054f9e47-675f-4722-b743-79bbdc826585\") " pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" Oct 31 01:11:40.032753 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:40.032726 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz4kp\" (UniqueName: \"kubernetes.io/projected/054f9e47-675f-4722-b743-79bbdc826585-kube-api-access-bz4kp\") pod \"distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg\" (UID: \"054f9e47-675f-4722-b743-79bbdc826585\") " pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" Oct 31 01:11:40.134476 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:40.134433 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/054f9e47-675f-4722-b743-79bbdc826585-tls-certs\") pod \"distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg\" (UID: \"054f9e47-675f-4722-b743-79bbdc826585\") " pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" Oct 31 01:11:40.134657 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:40.134486 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/054f9e47-675f-4722-b743-79bbdc826585-model-cache\") pod \"distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg\" (UID: \"054f9e47-675f-4722-b743-79bbdc826585\") " pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" Oct 31 01:11:40.134657 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:40.134550 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/054f9e47-675f-4722-b743-79bbdc826585-home\") pod \"distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg\" (UID: \"054f9e47-675f-4722-b743-79bbdc826585\") " pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" Oct 31 01:11:40.134657 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:40.134580 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/054f9e47-675f-4722-b743-79bbdc826585-tmp-dir\") pod \"distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg\" (UID: \"054f9e47-675f-4722-b743-79bbdc826585\") " pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" Oct 31 01:11:40.134657 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:40.134651 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz4kp\" (UniqueName: \"kubernetes.io/projected/054f9e47-675f-4722-b743-79bbdc826585-kube-api-access-bz4kp\") pod \"distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg\" (UID: \"054f9e47-675f-4722-b743-79bbdc826585\") " pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" Oct 31 01:11:40.134818 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:40.134772 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/054f9e47-675f-4722-b743-79bbdc826585-dshm\") pod \"distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg\" (UID: \"054f9e47-675f-4722-b743-79bbdc826585\") " pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" Oct 31 01:11:40.134956 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:40.134928 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/054f9e47-675f-4722-b743-79bbdc826585-model-cache\") pod \"distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg\" (UID: \"054f9e47-675f-4722-b743-79bbdc826585\") " pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" Oct 31 01:11:40.135279 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:40.135246 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/054f9e47-675f-4722-b743-79bbdc826585-tmp-dir\") pod \"distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg\" (UID: \"054f9e47-675f-4722-b743-79bbdc826585\") " pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" Oct 31 01:11:40.135558 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:40.135533 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/054f9e47-675f-4722-b743-79bbdc826585-home\") pod \"distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg\" (UID: \"054f9e47-675f-4722-b743-79bbdc826585\") " pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" Oct 31 01:11:40.137704 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:40.137678 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/054f9e47-675f-4722-b743-79bbdc826585-tls-certs\") pod \"distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg\" (UID: \"054f9e47-675f-4722-b743-79bbdc826585\") " pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" Oct 31 01:11:40.137899 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:40.137881 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/054f9e47-675f-4722-b743-79bbdc826585-dshm\") pod \"distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg\" (UID: \"054f9e47-675f-4722-b743-79bbdc826585\") " pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" Oct 31 01:11:40.144287 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:40.144260 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz4kp\" (UniqueName: \"kubernetes.io/projected/054f9e47-675f-4722-b743-79bbdc826585-kube-api-access-bz4kp\") pod \"distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg\" (UID: \"054f9e47-675f-4722-b743-79bbdc826585\") " pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" Oct 31 01:11:40.258834 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:40.258746 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" Oct 31 01:11:40.513525 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:40.513407 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:40.553328 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:40.553295 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg"] Oct 31 01:11:40.557542 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:11:40.557479 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod054f9e47_675f_4722_b743_79bbdc826585.slice/crio-f6e00286ab6fa244240117d3a981d1c5e36e193d853adbb39a7632b12bb35698 WatchSource:0}: Error finding container f6e00286ab6fa244240117d3a981d1c5e36e193d853adbb39a7632b12bb35698: Status 404 returned error can't find the container with id f6e00286ab6fa244240117d3a981d1c5e36e193d853adbb39a7632b12bb35698 Oct 31 01:11:40.640330 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:40.640288 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:41.516749 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:41.516715 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" event={"ID":"054f9e47-675f-4722-b743-79bbdc826585","Type":"ContainerStarted","Data":"a011701ce2a965b13374b7fe0d5c19a20d52c7217b49af41afd83e75c8f875c4"} Oct 31 01:11:41.517129 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:41.516760 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" event={"ID":"054f9e47-675f-4722-b743-79bbdc826585","Type":"ContainerStarted","Data":"f6e00286ab6fa244240117d3a981d1c5e36e193d853adbb39a7632b12bb35698"} Oct 31 01:11:41.533851 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:41.533801 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podStartSLOduration=2.533784475 podStartE2EDuration="2.533784475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:11:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:11:41.532403453 +0000 UTC m=+945.891125769" watchObservedRunningTime="2026-10-31 01:11:41.533784475 +0000 UTC m=+945.892506795" Oct 31 01:11:42.014696 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:42.014640 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:42.259767 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:42.259732 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" Oct 31 01:11:42.973660 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:42.973626 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:44.056680 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:44.056644 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:44.854371 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:44.854335 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:45.841801 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:45.841754 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:45.962399 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:45.962361 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:47.347216 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:47.345839 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:48.395447 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:48.394504 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:11:48.458447 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:48.457522 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 01:11:48.525664 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:48.525620 4440 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 01:11:48.793091 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:48.792508 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:11:49.042558 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:49.042520 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:49.693688 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:49.693653 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:49.754333 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:49.754287 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj"] Oct 31 01:11:49.760620 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:49.759531 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" Oct 31 01:11:49.774003 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:49.773978 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-1b8d05\"/\"distinct-2-w6-1b8d05-kserve-self-signed-certs\"" Oct 31 01:11:49.779938 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:49.779913 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj"] Oct 31 01:11:49.828112 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:49.828070 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:49.938647 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:49.938599 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c29662e2-8234-4786-9b42-30ca828deb74-dshm\") pod \"distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj\" (UID: \"c29662e2-8234-4786-9b42-30ca828deb74\") " pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" Oct 31 01:11:49.938964 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:49.938942 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vbr4\" (UniqueName: \"kubernetes.io/projected/c29662e2-8234-4786-9b42-30ca828deb74-kube-api-access-6vbr4\") pod \"distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj\" (UID: \"c29662e2-8234-4786-9b42-30ca828deb74\") " pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" Oct 31 01:11:49.939144 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:49.939128 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c29662e2-8234-4786-9b42-30ca828deb74-home\") pod \"distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj\" (UID: \"c29662e2-8234-4786-9b42-30ca828deb74\") " pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" Oct 31 01:11:49.944056 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:49.940088 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c29662e2-8234-4786-9b42-30ca828deb74-model-cache\") pod \"distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj\" (UID: \"c29662e2-8234-4786-9b42-30ca828deb74\") " pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" Oct 31 01:11:49.944324 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:49.944304 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c29662e2-8234-4786-9b42-30ca828deb74-tmp-dir\") pod \"distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj\" (UID: \"c29662e2-8234-4786-9b42-30ca828deb74\") " pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" Oct 31 01:11:49.944528 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:49.944494 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c29662e2-8234-4786-9b42-30ca828deb74-tls-certs\") pod \"distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj\" (UID: \"c29662e2-8234-4786-9b42-30ca828deb74\") " pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" Oct 31 01:11:50.046393 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:50.046354 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c29662e2-8234-4786-9b42-30ca828deb74-model-cache\") pod \"distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj\" (UID: \"c29662e2-8234-4786-9b42-30ca828deb74\") " pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" Oct 31 01:11:50.046725 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:50.046696 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c29662e2-8234-4786-9b42-30ca828deb74-tmp-dir\") pod \"distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj\" (UID: \"c29662e2-8234-4786-9b42-30ca828deb74\") " pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" Oct 31 01:11:50.046854 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:50.046743 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c29662e2-8234-4786-9b42-30ca828deb74-tls-certs\") pod \"distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj\" (UID: \"c29662e2-8234-4786-9b42-30ca828deb74\") " pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" Oct 31 01:11:50.046854 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:50.046785 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c29662e2-8234-4786-9b42-30ca828deb74-dshm\") pod \"distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj\" (UID: \"c29662e2-8234-4786-9b42-30ca828deb74\") " pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" Oct 31 01:11:50.046980 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:50.046858 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vbr4\" (UniqueName: \"kubernetes.io/projected/c29662e2-8234-4786-9b42-30ca828deb74-kube-api-access-6vbr4\") pod \"distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj\" (UID: \"c29662e2-8234-4786-9b42-30ca828deb74\") " pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" Oct 31 01:11:50.046980 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:50.046890 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c29662e2-8234-4786-9b42-30ca828deb74-model-cache\") pod \"distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj\" (UID: \"c29662e2-8234-4786-9b42-30ca828deb74\") " pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" Oct 31 01:11:50.047122 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:50.047099 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c29662e2-8234-4786-9b42-30ca828deb74-tmp-dir\") pod \"distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj\" (UID: \"c29662e2-8234-4786-9b42-30ca828deb74\") " pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" Oct 31 01:11:50.047179 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:50.047159 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c29662e2-8234-4786-9b42-30ca828deb74-home\") pod \"distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj\" (UID: \"c29662e2-8234-4786-9b42-30ca828deb74\") " pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" Oct 31 01:11:50.047497 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:50.047475 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c29662e2-8234-4786-9b42-30ca828deb74-home\") pod \"distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj\" (UID: \"c29662e2-8234-4786-9b42-30ca828deb74\") " pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" Oct 31 01:11:50.049747 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:50.049719 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c29662e2-8234-4786-9b42-30ca828deb74-dshm\") pod \"distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj\" (UID: \"c29662e2-8234-4786-9b42-30ca828deb74\") " pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" Oct 31 01:11:50.050008 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:50.049983 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c29662e2-8234-4786-9b42-30ca828deb74-tls-certs\") pod \"distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj\" (UID: \"c29662e2-8234-4786-9b42-30ca828deb74\") " pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" Oct 31 01:11:50.055561 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:50.055537 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vbr4\" (UniqueName: \"kubernetes.io/projected/c29662e2-8234-4786-9b42-30ca828deb74-kube-api-access-6vbr4\") pod \"distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj\" (UID: \"c29662e2-8234-4786-9b42-30ca828deb74\") " pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" Oct 31 01:11:50.083373 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:50.083348 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" Oct 31 01:11:50.259570 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:50.259278 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" Oct 31 01:11:50.277264 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:50.277225 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" Oct 31 01:11:50.395504 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:11:50.395457 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc29662e2_8234_4786_9b42_30ca828deb74.slice/crio-b2d9e159ddd92f632c353eb0bb43bf6f5b893237cde91601daa138f0f75e26b3 WatchSource:0}: Error finding container b2d9e159ddd92f632c353eb0bb43bf6f5b893237cde91601daa138f0f75e26b3: Status 404 returned error can't find the container with id b2d9e159ddd92f632c353eb0bb43bf6f5b893237cde91601daa138f0f75e26b3 Oct 31 01:11:50.405744 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:50.405702 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj"] Oct 31 01:11:50.476714 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:50.476636 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:50.633460 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:50.633358 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" event={"ID":"c29662e2-8234-4786-9b42-30ca828deb74","Type":"ContainerStarted","Data":"91cddc094a32c92a52ca7aea754d54b418916c6acf4e8e501e8f7cf99c150ca9"} Oct 31 01:11:50.633460 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:50.633411 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" event={"ID":"c29662e2-8234-4786-9b42-30ca828deb74","Type":"ContainerStarted","Data":"b2d9e159ddd92f632c353eb0bb43bf6f5b893237cde91601daa138f0f75e26b3"} Oct 31 01:11:50.645383 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:50.645353 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" Oct 31 01:11:50.651559 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:50.651516 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" podStartSLOduration=1.6515046070000001 podStartE2EDuration="1.651504607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:11:50.649694053 +0000 UTC m=+955.008416369" watchObservedRunningTime="2026-10-31 01:11:50.651504607 +0000 UTC m=+955.010226922" Oct 31 01:11:51.083690 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:51.083654 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" Oct 31 01:11:51.257325 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:51.257295 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:52.075229 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.075192 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf"] Oct 31 01:11:52.079341 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.079325 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" Oct 31 01:11:52.090933 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.090908 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf"] Oct 31 01:11:52.175250 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.175217 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-istio-token\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf\" (UID: \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" Oct 31 01:11:52.175447 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.175288 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-credential-socket\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf\" (UID: \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" Oct 31 01:11:52.175447 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.175362 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-istio-envoy\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf\" (UID: \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" Oct 31 01:11:52.175447 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.175392 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-workload-socket\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf\" (UID: \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" Oct 31 01:11:52.175586 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.175491 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-istiod-ca-cert\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf\" (UID: \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" Oct 31 01:11:52.175586 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.175519 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-istio-podinfo\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf\" (UID: \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" Oct 31 01:11:52.175586 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.175536 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-workload-certs\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf\" (UID: \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" Oct 31 01:11:52.175695 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.175659 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xfpx\" (UniqueName: \"kubernetes.io/projected/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-kube-api-access-8xfpx\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf\" (UID: \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" Oct 31 01:11:52.175695 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.175689 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-istio-data\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf\" (UID: \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" Oct 31 01:11:52.234190 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.234152 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:52.276530 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.276492 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-istio-data\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf\" (UID: \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" Oct 31 01:11:52.276723 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.276612 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-istio-token\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf\" (UID: \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" Oct 31 01:11:52.276723 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.276650 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-credential-socket\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf\" (UID: \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" Oct 31 01:11:52.276723 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.276707 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-istio-envoy\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf\" (UID: \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" Oct 31 01:11:52.276889 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.276735 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-workload-socket\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf\" (UID: \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" Oct 31 01:11:52.276889 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.276779 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-istiod-ca-cert\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf\" (UID: \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" Oct 31 01:11:52.276889 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.276817 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-istio-podinfo\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf\" (UID: \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" Oct 31 01:11:52.277042 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.276961 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-istio-data\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf\" (UID: \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" Oct 31 01:11:52.277042 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.277014 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-workload-certs\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf\" (UID: \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" Oct 31 01:11:52.277157 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.277134 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xfpx\" (UniqueName: \"kubernetes.io/projected/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-kube-api-access-8xfpx\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf\" (UID: \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" Oct 31 01:11:52.277223 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.277199 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-workload-socket\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf\" (UID: \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" Oct 31 01:11:52.277491 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.277468 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-workload-certs\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf\" (UID: \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" Oct 31 01:11:52.277619 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.277510 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-credential-socket\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf\" (UID: \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" Oct 31 01:11:52.277868 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.277843 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-istiod-ca-cert\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf\" (UID: \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" Oct 31 01:11:52.280580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.279564 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-istio-envoy\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf\" (UID: \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" Oct 31 01:11:52.280580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.279944 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-istio-podinfo\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf\" (UID: \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" Oct 31 01:11:52.285947 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.285900 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-istio-token\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf\" (UID: \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" Oct 31 01:11:52.286939 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.286914 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xfpx\" (UniqueName: \"kubernetes.io/projected/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-kube-api-access-8xfpx\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf\" (UID: \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" Oct 31 01:11:52.395886 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.395791 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" Oct 31 01:11:52.688518 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.688493 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf"] Oct 31 01:11:52.691405 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:11:52.691384 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a9a9349_7fdf_4f3a_8f96_62c9134ba1a5.slice/crio-d4cb61f9f229aeff75897b408c7fd4ee9dbdb8a2affa9bfef64e19c426519c49 WatchSource:0}: Error finding container d4cb61f9f229aeff75897b408c7fd4ee9dbdb8a2affa9bfef64e19c426519c49: Status 404 returned error can't find the container with id d4cb61f9f229aeff75897b408c7fd4ee9dbdb8a2affa9bfef64e19c426519c49 Oct 31 01:11:52.693719 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.693691 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:11:52.693790 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.693752 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:11:52.693790 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.693782 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:11:52.766810 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.766777 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv"] Oct 31 01:11:52.772058 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.772038 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" Oct 31 01:11:52.780866 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.780841 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv"] Oct 31 01:11:52.783069 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.783044 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-0a6cca\"/\"distinct-w2-0a6cca-kserve-self-signed-certs\"" Oct 31 01:11:52.786653 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.786628 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2ca4946-eb55-4a6b-a4bb-257a5792ffed-tmp-dir\") pod \"distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv\" (UID: \"f2ca4946-eb55-4a6b-a4bb-257a5792ffed\") " pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" Oct 31 01:11:52.786764 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.786713 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2ca4946-eb55-4a6b-a4bb-257a5792ffed-model-cache\") pod \"distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv\" (UID: \"f2ca4946-eb55-4a6b-a4bb-257a5792ffed\") " pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" Oct 31 01:11:52.786764 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.786752 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2ca4946-eb55-4a6b-a4bb-257a5792ffed-home\") pod \"distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv\" (UID: \"f2ca4946-eb55-4a6b-a4bb-257a5792ffed\") " pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" Oct 31 01:11:52.786869 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.786834 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2ca4946-eb55-4a6b-a4bb-257a5792ffed-tls-certs\") pod \"distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv\" (UID: \"f2ca4946-eb55-4a6b-a4bb-257a5792ffed\") " pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" Oct 31 01:11:52.787329 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.787041 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfj7l\" (UniqueName: \"kubernetes.io/projected/f2ca4946-eb55-4a6b-a4bb-257a5792ffed-kube-api-access-cfj7l\") pod \"distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv\" (UID: \"f2ca4946-eb55-4a6b-a4bb-257a5792ffed\") " pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" Oct 31 01:11:52.787329 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.787134 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2ca4946-eb55-4a6b-a4bb-257a5792ffed-dshm\") pod \"distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv\" (UID: \"f2ca4946-eb55-4a6b-a4bb-257a5792ffed\") " pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" Oct 31 01:11:52.846985 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.846927 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:52.888052 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.888017 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfj7l\" (UniqueName: \"kubernetes.io/projected/f2ca4946-eb55-4a6b-a4bb-257a5792ffed-kube-api-access-cfj7l\") pod \"distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv\" (UID: \"f2ca4946-eb55-4a6b-a4bb-257a5792ffed\") " pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" Oct 31 01:11:52.888255 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.888076 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2ca4946-eb55-4a6b-a4bb-257a5792ffed-dshm\") pod \"distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv\" (UID: \"f2ca4946-eb55-4a6b-a4bb-257a5792ffed\") " pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" Oct 31 01:11:52.888255 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.888131 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2ca4946-eb55-4a6b-a4bb-257a5792ffed-tmp-dir\") pod \"distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv\" (UID: \"f2ca4946-eb55-4a6b-a4bb-257a5792ffed\") " pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" Oct 31 01:11:52.888255 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.888221 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2ca4946-eb55-4a6b-a4bb-257a5792ffed-model-cache\") pod \"distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv\" (UID: \"f2ca4946-eb55-4a6b-a4bb-257a5792ffed\") " pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" Oct 31 01:11:52.888409 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.888258 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2ca4946-eb55-4a6b-a4bb-257a5792ffed-home\") pod \"distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv\" (UID: \"f2ca4946-eb55-4a6b-a4bb-257a5792ffed\") " pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" Oct 31 01:11:52.888409 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.888300 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2ca4946-eb55-4a6b-a4bb-257a5792ffed-tls-certs\") pod \"distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv\" (UID: \"f2ca4946-eb55-4a6b-a4bb-257a5792ffed\") " pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" Oct 31 01:11:52.888677 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.888657 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2ca4946-eb55-4a6b-a4bb-257a5792ffed-tmp-dir\") pod \"distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv\" (UID: \"f2ca4946-eb55-4a6b-a4bb-257a5792ffed\") " pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" Oct 31 01:11:52.888787 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.888764 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2ca4946-eb55-4a6b-a4bb-257a5792ffed-home\") pod \"distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv\" (UID: \"f2ca4946-eb55-4a6b-a4bb-257a5792ffed\") " pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" Oct 31 01:11:52.889037 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.889015 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2ca4946-eb55-4a6b-a4bb-257a5792ffed-model-cache\") pod \"distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv\" (UID: \"f2ca4946-eb55-4a6b-a4bb-257a5792ffed\") " pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" Oct 31 01:11:52.890467 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.890407 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2ca4946-eb55-4a6b-a4bb-257a5792ffed-dshm\") pod \"distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv\" (UID: \"f2ca4946-eb55-4a6b-a4bb-257a5792ffed\") " pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" Oct 31 01:11:52.890709 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.890692 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2ca4946-eb55-4a6b-a4bb-257a5792ffed-tls-certs\") pod \"distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv\" (UID: \"f2ca4946-eb55-4a6b-a4bb-257a5792ffed\") " pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" Oct 31 01:11:52.895904 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:52.895884 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfj7l\" (UniqueName: \"kubernetes.io/projected/f2ca4946-eb55-4a6b-a4bb-257a5792ffed-kube-api-access-cfj7l\") pod \"distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv\" (UID: \"f2ca4946-eb55-4a6b-a4bb-257a5792ffed\") " pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" Oct 31 01:11:53.089477 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:53.089340 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" Oct 31 01:11:53.334563 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:53.329836 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:53.375398 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:53.375316 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv"] Oct 31 01:11:53.379469 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:11:53.379431 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2ca4946_eb55_4a6b_a4bb_257a5792ffed.slice/crio-7746e8bedb4d1977ac24b0f2e11d87eed164e32e112584ccab7ca498c2587877 WatchSource:0}: Error finding container 7746e8bedb4d1977ac24b0f2e11d87eed164e32e112584ccab7ca498c2587877: Status 404 returned error can't find the container with id 7746e8bedb4d1977ac24b0f2e11d87eed164e32e112584ccab7ca498c2587877 Oct 31 01:11:53.662865 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:53.662777 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" event={"ID":"f2ca4946-eb55-4a6b-a4bb-257a5792ffed","Type":"ContainerStarted","Data":"7ab14572d848ec37a73de36151f934ccb2da98f4021a7dca143227067ade51be"} Oct 31 01:11:53.662865 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:53.662826 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" event={"ID":"f2ca4946-eb55-4a6b-a4bb-257a5792ffed","Type":"ContainerStarted","Data":"7746e8bedb4d1977ac24b0f2e11d87eed164e32e112584ccab7ca498c2587877"} Oct 31 01:11:53.664637 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:53.664603 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" event={"ID":"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5","Type":"ContainerStarted","Data":"21e155795263856e2b406f7d54f3d4d6dcff08395039e8dc0b1f047a058c2b56"} Oct 31 01:11:53.664769 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:53.664700 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" event={"ID":"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5","Type":"ContainerStarted","Data":"d4cb61f9f229aeff75897b408c7fd4ee9dbdb8a2affa9bfef64e19c426519c49"} Oct 31 01:11:53.682410 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:53.682352 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podStartSLOduration=1.6823406319999998 podStartE2EDuration="1.682340632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:11:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:11:53.680674156 +0000 UTC m=+958.039396471" watchObservedRunningTime="2026-10-31 01:11:53.682340632 +0000 UTC m=+958.041062957" Oct 31 01:11:54.090120 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:54.090088 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" Oct 31 01:11:54.131508 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:54.128034 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:54.399500 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:54.396791 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" Oct 31 01:11:54.403445 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:54.402534 4440 patch_prober.go:29] interesting pod/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 01:11:54.403445 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:54.402598 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" podUID="0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 01:11:54.945478 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:54.945401 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:55.396590 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:55.396485 4440 patch_prober.go:29] interesting pod/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 01:11:55.396877 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:55.396831 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" podUID="0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 01:11:55.826577 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:55.826538 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:56.402445 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:56.399796 4440 patch_prober.go:29] interesting pod/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 01:11:56.402445 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:56.399860 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" podUID="0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 01:11:56.522292 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:56.522192 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:57.266689 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:57.266644 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:57.403444 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:57.399748 4440 patch_prober.go:29] interesting pod/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 01:11:57.403444 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:57.399814 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" podUID="0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 01:11:58.023362 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:58.023311 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:58.131293 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:58.131256 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:58.400493 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:58.399750 4440 patch_prober.go:29] interesting pod/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 01:11:58.400493 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:58.399812 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" podUID="0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 01:11:58.731009 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:58.730951 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:59.464297 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:59.464263 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:59.653745 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:59.653705 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" Oct 31 01:11:59.655192 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:59.655170 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" Oct 31 01:11:59.656030 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:59.656012 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" Oct 31 01:11:59.685243 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:59.685179 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" podStartSLOduration=7.685158071 podStartE2EDuration="7.685158071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:11:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:11:53.723057245 +0000 UTC m=+958.081779562" watchObservedRunningTime="2026-10-31 01:11:59.685158071 +0000 UTC m=+964.043880390" Oct 31 01:11:59.791355 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:59.790592 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw"] Oct 31 01:11:59.798602 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:59.798577 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" Oct 31 01:11:59.809748 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:59.809720 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-28ff08\"/\"kube-root-ca.crt\"" Oct 31 01:11:59.810006 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:59.809987 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-28ff08\"/\"openshift-service-ca.crt\"" Oct 31 01:11:59.810339 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:59.810317 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw"] Oct 31 01:11:59.826070 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:59.826042 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-28ff08\"/\"default-dockercfg-6kts9\"" Oct 31 01:11:59.826193 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:59.826182 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-28ff08\"/\"distinct-w5-28ff08-kserve-self-signed-certs\"" Oct 31 01:11:59.868237 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:59.868208 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:59.897946 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:59.897918 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f93d4671-a358-4547-ba77-06255ca42c3e-tls-certs\") pod \"distinct-w5-28ff08-kserve-767868b958-zszvw\" (UID: \"f93d4671-a358-4547-ba77-06255ca42c3e\") " pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" Oct 31 01:11:59.898055 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:59.897973 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk4mt\" (UniqueName: \"kubernetes.io/projected/f93d4671-a358-4547-ba77-06255ca42c3e-kube-api-access-qk4mt\") pod \"distinct-w5-28ff08-kserve-767868b958-zszvw\" (UID: \"f93d4671-a358-4547-ba77-06255ca42c3e\") " pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" Oct 31 01:11:59.898055 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:59.898033 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f93d4671-a358-4547-ba77-06255ca42c3e-model-cache\") pod \"distinct-w5-28ff08-kserve-767868b958-zszvw\" (UID: \"f93d4671-a358-4547-ba77-06255ca42c3e\") " pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" Oct 31 01:11:59.898138 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:59.898059 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f93d4671-a358-4547-ba77-06255ca42c3e-dshm\") pod \"distinct-w5-28ff08-kserve-767868b958-zszvw\" (UID: \"f93d4671-a358-4547-ba77-06255ca42c3e\") " pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" Oct 31 01:11:59.898138 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:59.898075 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f93d4671-a358-4547-ba77-06255ca42c3e-tmp-dir\") pod \"distinct-w5-28ff08-kserve-767868b958-zszvw\" (UID: \"f93d4671-a358-4547-ba77-06255ca42c3e\") " pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" Oct 31 01:11:59.898237 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:59.898219 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f93d4671-a358-4547-ba77-06255ca42c3e-home\") pod \"distinct-w5-28ff08-kserve-767868b958-zszvw\" (UID: \"f93d4671-a358-4547-ba77-06255ca42c3e\") " pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" Oct 31 01:12:00.016473 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:59.999805 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f93d4671-a358-4547-ba77-06255ca42c3e-home\") pod \"distinct-w5-28ff08-kserve-767868b958-zszvw\" (UID: \"f93d4671-a358-4547-ba77-06255ca42c3e\") " pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" Oct 31 01:12:00.016473 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:11:59.999982 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f93d4671-a358-4547-ba77-06255ca42c3e-tls-certs\") pod \"distinct-w5-28ff08-kserve-767868b958-zszvw\" (UID: \"f93d4671-a358-4547-ba77-06255ca42c3e\") " pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" Oct 31 01:12:00.016473 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:00.000051 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk4mt\" (UniqueName: \"kubernetes.io/projected/f93d4671-a358-4547-ba77-06255ca42c3e-kube-api-access-qk4mt\") pod \"distinct-w5-28ff08-kserve-767868b958-zszvw\" (UID: \"f93d4671-a358-4547-ba77-06255ca42c3e\") " pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" Oct 31 01:12:00.016473 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:00.000082 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f93d4671-a358-4547-ba77-06255ca42c3e-model-cache\") pod \"distinct-w5-28ff08-kserve-767868b958-zszvw\" (UID: \"f93d4671-a358-4547-ba77-06255ca42c3e\") " pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" Oct 31 01:12:00.016473 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:00.000119 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f93d4671-a358-4547-ba77-06255ca42c3e-dshm\") pod \"distinct-w5-28ff08-kserve-767868b958-zszvw\" (UID: \"f93d4671-a358-4547-ba77-06255ca42c3e\") " pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" Oct 31 01:12:00.016473 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:00.000139 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f93d4671-a358-4547-ba77-06255ca42c3e-tmp-dir\") pod \"distinct-w5-28ff08-kserve-767868b958-zszvw\" (UID: \"f93d4671-a358-4547-ba77-06255ca42c3e\") " pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" Oct 31 01:12:00.016473 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:00.000652 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f93d4671-a358-4547-ba77-06255ca42c3e-tmp-dir\") pod \"distinct-w5-28ff08-kserve-767868b958-zszvw\" (UID: \"f93d4671-a358-4547-ba77-06255ca42c3e\") " pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" Oct 31 01:12:00.016473 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:00.000905 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f93d4671-a358-4547-ba77-06255ca42c3e-home\") pod \"distinct-w5-28ff08-kserve-767868b958-zszvw\" (UID: \"f93d4671-a358-4547-ba77-06255ca42c3e\") " pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" Oct 31 01:12:00.016473 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:00.005808 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f93d4671-a358-4547-ba77-06255ca42c3e-tls-certs\") pod \"distinct-w5-28ff08-kserve-767868b958-zszvw\" (UID: \"f93d4671-a358-4547-ba77-06255ca42c3e\") " pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" Oct 31 01:12:00.016473 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:00.006393 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f93d4671-a358-4547-ba77-06255ca42c3e-model-cache\") pod \"distinct-w5-28ff08-kserve-767868b958-zszvw\" (UID: \"f93d4671-a358-4547-ba77-06255ca42c3e\") " pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" Oct 31 01:12:00.024447 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:00.023847 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f93d4671-a358-4547-ba77-06255ca42c3e-dshm\") pod \"distinct-w5-28ff08-kserve-767868b958-zszvw\" (UID: \"f93d4671-a358-4547-ba77-06255ca42c3e\") " pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" Oct 31 01:12:00.027877 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:00.027820 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk4mt\" (UniqueName: \"kubernetes.io/projected/f93d4671-a358-4547-ba77-06255ca42c3e-kube-api-access-qk4mt\") pod \"distinct-w5-28ff08-kserve-767868b958-zszvw\" (UID: \"f93d4671-a358-4547-ba77-06255ca42c3e\") " pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" Oct 31 01:12:00.084151 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:00.084050 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" Oct 31 01:12:00.097663 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:00.097633 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" Oct 31 01:12:00.143441 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:00.122050 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" Oct 31 01:12:00.436743 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:00.436710 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw"] Oct 31 01:12:00.437222 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:12:00.437199 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf93d4671_a358_4547_ba77_06255ca42c3e.slice/crio-0b497359e9a3e33043c1cd6231efda11de162feef4dd3688908af576d8ffc369 WatchSource:0}: Error finding container 0b497359e9a3e33043c1cd6231efda11de162feef4dd3688908af576d8ffc369: Status 404 returned error can't find the container with id 0b497359e9a3e33043c1cd6231efda11de162feef4dd3688908af576d8ffc369 Oct 31 01:12:00.626257 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:00.626220 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:00.721208 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:00.721127 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" event={"ID":"f93d4671-a358-4547-ba77-06255ca42c3e","Type":"ContainerStarted","Data":"c3ff32cc0a574168dc30068c8dcc6454aaa719461d84018cce68768fdc6b1b78"} Oct 31 01:12:00.721208 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:00.721169 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" event={"ID":"f93d4671-a358-4547-ba77-06255ca42c3e","Type":"ContainerStarted","Data":"0b497359e9a3e33043c1cd6231efda11de162feef4dd3688908af576d8ffc369"} Oct 31 01:12:00.739005 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:00.738978 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" Oct 31 01:12:00.757025 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:00.756984 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podStartSLOduration=1.756972111 podStartE2EDuration="1.756972111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:00.754599397 +0000 UTC m=+965.113321712" watchObservedRunningTime="2026-10-31 01:12:00.756972111 +0000 UTC m=+965.115694489" Oct 31 01:12:01.021514 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:12:01.020791 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:12:01.122989 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:01.122957 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" Oct 31 01:12:01.559618 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:01.559584 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:01.687102 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:01.687064 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:02.011071 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:02.011029 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:12:02.457492 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:02.457455 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:03.089928 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:03.089891 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" Oct 31 01:12:03.106788 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:03.106761 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" Oct 31 01:12:03.846505 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:03.846456 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:04.603575 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:04.601821 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" Oct 31 01:12:04.997572 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:04.997497 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:05.699649 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:05.699606 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:06.410134 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:06.410092 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:06.618476 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:06.618445 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq"] Oct 31 01:12:06.623335 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:06.623315 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" Oct 31 01:12:06.635097 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:06.635071 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq"] Oct 31 01:12:06.765776 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:06.765739 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb48867-402c-40ca-8db3-c513636cbf20-workload-certs\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq\" (UID: \"4eb48867-402c-40ca-8db3-c513636cbf20\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" Oct 31 01:12:06.766222 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:06.765782 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb48867-402c-40ca-8db3-c513636cbf20-workload-socket\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq\" (UID: \"4eb48867-402c-40ca-8db3-c513636cbf20\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" Oct 31 01:12:06.766222 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:06.765811 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eb48867-402c-40ca-8db3-c513636cbf20-istio-podinfo\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq\" (UID: \"4eb48867-402c-40ca-8db3-c513636cbf20\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" Oct 31 01:12:06.766222 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:06.765846 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eb48867-402c-40ca-8db3-c513636cbf20-istio-token\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq\" (UID: \"4eb48867-402c-40ca-8db3-c513636cbf20\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" Oct 31 01:12:06.766222 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:06.765875 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eb48867-402c-40ca-8db3-c513636cbf20-istio-envoy\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq\" (UID: \"4eb48867-402c-40ca-8db3-c513636cbf20\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" Oct 31 01:12:06.766222 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:06.765963 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzm4z\" (UniqueName: \"kubernetes.io/projected/4eb48867-402c-40ca-8db3-c513636cbf20-kube-api-access-jzm4z\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq\" (UID: \"4eb48867-402c-40ca-8db3-c513636cbf20\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" Oct 31 01:12:06.766222 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:06.766013 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eb48867-402c-40ca-8db3-c513636cbf20-istio-data\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq\" (UID: \"4eb48867-402c-40ca-8db3-c513636cbf20\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" Oct 31 01:12:06.766222 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:06.766164 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb48867-402c-40ca-8db3-c513636cbf20-credential-socket\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq\" (UID: \"4eb48867-402c-40ca-8db3-c513636cbf20\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" Oct 31 01:12:06.766222 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:06.766190 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eb48867-402c-40ca-8db3-c513636cbf20-istiod-ca-cert\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq\" (UID: \"4eb48867-402c-40ca-8db3-c513636cbf20\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" Oct 31 01:12:06.867025 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:06.866986 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eb48867-402c-40ca-8db3-c513636cbf20-istio-data\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq\" (UID: \"4eb48867-402c-40ca-8db3-c513636cbf20\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" Oct 31 01:12:06.867206 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:06.867152 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb48867-402c-40ca-8db3-c513636cbf20-credential-socket\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq\" (UID: \"4eb48867-402c-40ca-8db3-c513636cbf20\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" Oct 31 01:12:06.867206 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:06.867185 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eb48867-402c-40ca-8db3-c513636cbf20-istiod-ca-cert\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq\" (UID: \"4eb48867-402c-40ca-8db3-c513636cbf20\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" Oct 31 01:12:06.867339 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:06.867316 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb48867-402c-40ca-8db3-c513636cbf20-workload-certs\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq\" (UID: \"4eb48867-402c-40ca-8db3-c513636cbf20\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" Oct 31 01:12:06.867396 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:06.867349 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb48867-402c-40ca-8db3-c513636cbf20-workload-socket\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq\" (UID: \"4eb48867-402c-40ca-8db3-c513636cbf20\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" Oct 31 01:12:06.867396 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:06.867375 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eb48867-402c-40ca-8db3-c513636cbf20-istio-podinfo\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq\" (UID: \"4eb48867-402c-40ca-8db3-c513636cbf20\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" Oct 31 01:12:06.867553 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:06.867411 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eb48867-402c-40ca-8db3-c513636cbf20-istio-token\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq\" (UID: \"4eb48867-402c-40ca-8db3-c513636cbf20\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" Oct 31 01:12:06.867553 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:06.867475 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eb48867-402c-40ca-8db3-c513636cbf20-istio-envoy\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq\" (UID: \"4eb48867-402c-40ca-8db3-c513636cbf20\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" Oct 31 01:12:06.867553 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:06.867500 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eb48867-402c-40ca-8db3-c513636cbf20-istio-data\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq\" (UID: \"4eb48867-402c-40ca-8db3-c513636cbf20\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" Oct 31 01:12:06.867553 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:06.867532 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzm4z\" (UniqueName: \"kubernetes.io/projected/4eb48867-402c-40ca-8db3-c513636cbf20-kube-api-access-jzm4z\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq\" (UID: \"4eb48867-402c-40ca-8db3-c513636cbf20\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" Oct 31 01:12:06.867762 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:06.867592 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb48867-402c-40ca-8db3-c513636cbf20-credential-socket\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq\" (UID: \"4eb48867-402c-40ca-8db3-c513636cbf20\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" Oct 31 01:12:06.867762 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:06.867639 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb48867-402c-40ca-8db3-c513636cbf20-workload-certs\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq\" (UID: \"4eb48867-402c-40ca-8db3-c513636cbf20\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" Oct 31 01:12:06.867762 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:06.867712 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb48867-402c-40ca-8db3-c513636cbf20-workload-socket\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq\" (UID: \"4eb48867-402c-40ca-8db3-c513636cbf20\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" Oct 31 01:12:06.867999 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:06.867980 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eb48867-402c-40ca-8db3-c513636cbf20-istiod-ca-cert\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq\" (UID: \"4eb48867-402c-40ca-8db3-c513636cbf20\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" Oct 31 01:12:06.869846 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:06.869817 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eb48867-402c-40ca-8db3-c513636cbf20-istio-envoy\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq\" (UID: \"4eb48867-402c-40ca-8db3-c513636cbf20\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" Oct 31 01:12:06.870019 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:06.869997 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eb48867-402c-40ca-8db3-c513636cbf20-istio-podinfo\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq\" (UID: \"4eb48867-402c-40ca-8db3-c513636cbf20\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" Oct 31 01:12:06.879902 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:06.879877 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eb48867-402c-40ca-8db3-c513636cbf20-istio-token\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq\" (UID: \"4eb48867-402c-40ca-8db3-c513636cbf20\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" Oct 31 01:12:06.879995 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:06.879941 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzm4z\" (UniqueName: \"kubernetes.io/projected/4eb48867-402c-40ca-8db3-c513636cbf20-kube-api-access-jzm4z\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq\" (UID: \"4eb48867-402c-40ca-8db3-c513636cbf20\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" Oct 31 01:12:06.918507 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:06.918477 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw"] Oct 31 01:12:06.929235 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:06.928791 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" Oct 31 01:12:06.937316 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:06.934406 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw"] Oct 31 01:12:06.945351 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:06.945326 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" Oct 31 01:12:07.009956 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.009914 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts"] Oct 31 01:12:07.018411 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.018343 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" Oct 31 01:12:07.039178 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.038972 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts"] Oct 31 01:12:07.052550 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.051507 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw"] Oct 31 01:12:07.066789 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.066647 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" Oct 31 01:12:07.072405 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.070849 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hq9d\" (UniqueName: \"kubernetes.io/projected/613c5424-a9e2-43da-95aa-807e2b99ab2e-kube-api-access-8hq9d\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw\" (UID: \"613c5424-a9e2-43da-95aa-807e2b99ab2e\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" Oct 31 01:12:07.072405 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.070892 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/613c5424-a9e2-43da-95aa-807e2b99ab2e-istiod-ca-cert\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw\" (UID: \"613c5424-a9e2-43da-95aa-807e2b99ab2e\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" Oct 31 01:12:07.072405 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.070942 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/613c5424-a9e2-43da-95aa-807e2b99ab2e-istio-envoy\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw\" (UID: \"613c5424-a9e2-43da-95aa-807e2b99ab2e\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" Oct 31 01:12:07.072405 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.070975 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/613c5424-a9e2-43da-95aa-807e2b99ab2e-workload-socket\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw\" (UID: \"613c5424-a9e2-43da-95aa-807e2b99ab2e\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" Oct 31 01:12:07.072405 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.071050 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/613c5424-a9e2-43da-95aa-807e2b99ab2e-workload-certs\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw\" (UID: \"613c5424-a9e2-43da-95aa-807e2b99ab2e\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" Oct 31 01:12:07.072405 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.071122 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/613c5424-a9e2-43da-95aa-807e2b99ab2e-istio-token\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw\" (UID: \"613c5424-a9e2-43da-95aa-807e2b99ab2e\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" Oct 31 01:12:07.072405 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.071193 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/613c5424-a9e2-43da-95aa-807e2b99ab2e-istio-podinfo\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw\" (UID: \"613c5424-a9e2-43da-95aa-807e2b99ab2e\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" Oct 31 01:12:07.072405 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.071323 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/613c5424-a9e2-43da-95aa-807e2b99ab2e-credential-socket\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw\" (UID: \"613c5424-a9e2-43da-95aa-807e2b99ab2e\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" Oct 31 01:12:07.072405 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.071347 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/613c5424-a9e2-43da-95aa-807e2b99ab2e-istio-data\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw\" (UID: \"613c5424-a9e2-43da-95aa-807e2b99ab2e\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" Oct 31 01:12:07.077935 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.077911 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw"] Oct 31 01:12:07.172891 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.172856 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/613c5424-a9e2-43da-95aa-807e2b99ab2e-istio-podinfo\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw\" (UID: \"613c5424-a9e2-43da-95aa-807e2b99ab2e\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" Oct 31 01:12:07.173056 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.172928 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/823db35e-e37c-448d-a70d-96b5a218ac5e-istio-envoy\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw\" (UID: \"823db35e-e37c-448d-a70d-96b5a218ac5e\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" Oct 31 01:12:07.173056 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.172949 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/583a1183-7c84-40ce-aecb-9663f82a8a47-istio-data\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts\" (UID: \"583a1183-7c84-40ce-aecb-9663f82a8a47\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" Oct 31 01:12:07.173056 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.172972 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/823db35e-e37c-448d-a70d-96b5a218ac5e-workload-socket\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw\" (UID: \"823db35e-e37c-448d-a70d-96b5a218ac5e\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" Oct 31 01:12:07.173056 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.172994 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/823db35e-e37c-448d-a70d-96b5a218ac5e-workload-certs\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw\" (UID: \"823db35e-e37c-448d-a70d-96b5a218ac5e\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" Oct 31 01:12:07.173056 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.173047 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/823db35e-e37c-448d-a70d-96b5a218ac5e-istio-token\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw\" (UID: \"823db35e-e37c-448d-a70d-96b5a218ac5e\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" Oct 31 01:12:07.173248 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.173088 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/613c5424-a9e2-43da-95aa-807e2b99ab2e-credential-socket\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw\" (UID: \"613c5424-a9e2-43da-95aa-807e2b99ab2e\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" Oct 31 01:12:07.173248 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.173104 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/613c5424-a9e2-43da-95aa-807e2b99ab2e-istio-data\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw\" (UID: \"613c5424-a9e2-43da-95aa-807e2b99ab2e\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" Oct 31 01:12:07.173248 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.173125 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/583a1183-7c84-40ce-aecb-9663f82a8a47-workload-socket\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts\" (UID: \"583a1183-7c84-40ce-aecb-9663f82a8a47\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" Oct 31 01:12:07.173248 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.173144 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/583a1183-7c84-40ce-aecb-9663f82a8a47-istio-envoy\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts\" (UID: \"583a1183-7c84-40ce-aecb-9663f82a8a47\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" Oct 31 01:12:07.173402 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.173243 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/823db35e-e37c-448d-a70d-96b5a218ac5e-credential-socket\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw\" (UID: \"823db35e-e37c-448d-a70d-96b5a218ac5e\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" Oct 31 01:12:07.173402 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.173382 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hq9d\" (UniqueName: \"kubernetes.io/projected/613c5424-a9e2-43da-95aa-807e2b99ab2e-kube-api-access-8hq9d\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw\" (UID: \"613c5424-a9e2-43da-95aa-807e2b99ab2e\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" Oct 31 01:12:07.173531 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.173431 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qmrf\" (UniqueName: \"kubernetes.io/projected/823db35e-e37c-448d-a70d-96b5a218ac5e-kube-api-access-6qmrf\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw\" (UID: \"823db35e-e37c-448d-a70d-96b5a218ac5e\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" Oct 31 01:12:07.173531 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.173461 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zmhr\" (UniqueName: \"kubernetes.io/projected/583a1183-7c84-40ce-aecb-9663f82a8a47-kube-api-access-8zmhr\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts\" (UID: \"583a1183-7c84-40ce-aecb-9663f82a8a47\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" Oct 31 01:12:07.173531 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.173492 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/613c5424-a9e2-43da-95aa-807e2b99ab2e-istiod-ca-cert\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw\" (UID: \"613c5424-a9e2-43da-95aa-807e2b99ab2e\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" Oct 31 01:12:07.173531 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.173494 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/613c5424-a9e2-43da-95aa-807e2b99ab2e-istio-data\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw\" (UID: \"613c5424-a9e2-43da-95aa-807e2b99ab2e\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" Oct 31 01:12:07.173725 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.173532 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/613c5424-a9e2-43da-95aa-807e2b99ab2e-credential-socket\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw\" (UID: \"613c5424-a9e2-43da-95aa-807e2b99ab2e\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" Oct 31 01:12:07.173725 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.173621 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/583a1183-7c84-40ce-aecb-9663f82a8a47-workload-certs\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts\" (UID: \"583a1183-7c84-40ce-aecb-9663f82a8a47\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" Oct 31 01:12:07.173725 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.173667 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/583a1183-7c84-40ce-aecb-9663f82a8a47-istiod-ca-cert\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts\" (UID: \"583a1183-7c84-40ce-aecb-9663f82a8a47\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" Oct 31 01:12:07.173725 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.173701 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/613c5424-a9e2-43da-95aa-807e2b99ab2e-istio-envoy\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw\" (UID: \"613c5424-a9e2-43da-95aa-807e2b99ab2e\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" Oct 31 01:12:07.173959 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.173762 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/613c5424-a9e2-43da-95aa-807e2b99ab2e-workload-socket\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw\" (UID: \"613c5424-a9e2-43da-95aa-807e2b99ab2e\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" Oct 31 01:12:07.173959 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.173788 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/823db35e-e37c-448d-a70d-96b5a218ac5e-istio-podinfo\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw\" (UID: \"823db35e-e37c-448d-a70d-96b5a218ac5e\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" Oct 31 01:12:07.173959 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.173817 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/583a1183-7c84-40ce-aecb-9663f82a8a47-istio-podinfo\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts\" (UID: \"583a1183-7c84-40ce-aecb-9663f82a8a47\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" Oct 31 01:12:07.174127 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.173934 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/613c5424-a9e2-43da-95aa-807e2b99ab2e-workload-certs\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw\" (UID: \"613c5424-a9e2-43da-95aa-807e2b99ab2e\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" Oct 31 01:12:07.174127 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.173986 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/823db35e-e37c-448d-a70d-96b5a218ac5e-istiod-ca-cert\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw\" (UID: \"823db35e-e37c-448d-a70d-96b5a218ac5e\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" Oct 31 01:12:07.174127 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.174061 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/613c5424-a9e2-43da-95aa-807e2b99ab2e-istio-token\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw\" (UID: \"613c5424-a9e2-43da-95aa-807e2b99ab2e\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" Oct 31 01:12:07.174127 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.174069 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/613c5424-a9e2-43da-95aa-807e2b99ab2e-istiod-ca-cert\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw\" (UID: \"613c5424-a9e2-43da-95aa-807e2b99ab2e\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" Oct 31 01:12:07.174127 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.174106 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/583a1183-7c84-40ce-aecb-9663f82a8a47-credential-socket\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts\" (UID: \"583a1183-7c84-40ce-aecb-9663f82a8a47\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" Oct 31 01:12:07.174361 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.174144 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/823db35e-e37c-448d-a70d-96b5a218ac5e-istio-data\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw\" (UID: \"823db35e-e37c-448d-a70d-96b5a218ac5e\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" Oct 31 01:12:07.174361 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.174145 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/613c5424-a9e2-43da-95aa-807e2b99ab2e-workload-socket\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw\" (UID: \"613c5424-a9e2-43da-95aa-807e2b99ab2e\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" Oct 31 01:12:07.174361 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.174168 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/583a1183-7c84-40ce-aecb-9663f82a8a47-istio-token\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts\" (UID: \"583a1183-7c84-40ce-aecb-9663f82a8a47\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" Oct 31 01:12:07.174483 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.174367 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/613c5424-a9e2-43da-95aa-807e2b99ab2e-workload-certs\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw\" (UID: \"613c5424-a9e2-43da-95aa-807e2b99ab2e\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" Oct 31 01:12:07.175689 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.175668 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/613c5424-a9e2-43da-95aa-807e2b99ab2e-istio-podinfo\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw\" (UID: \"613c5424-a9e2-43da-95aa-807e2b99ab2e\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" Oct 31 01:12:07.176212 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.176192 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/613c5424-a9e2-43da-95aa-807e2b99ab2e-istio-envoy\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw\" (UID: \"613c5424-a9e2-43da-95aa-807e2b99ab2e\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" Oct 31 01:12:07.181276 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.181248 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/613c5424-a9e2-43da-95aa-807e2b99ab2e-istio-token\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw\" (UID: \"613c5424-a9e2-43da-95aa-807e2b99ab2e\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" Oct 31 01:12:07.181483 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.181464 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hq9d\" (UniqueName: \"kubernetes.io/projected/613c5424-a9e2-43da-95aa-807e2b99ab2e-kube-api-access-8hq9d\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw\" (UID: \"613c5424-a9e2-43da-95aa-807e2b99ab2e\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" Oct 31 01:12:07.257144 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.257113 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" Oct 31 01:12:07.275980 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.275951 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/823db35e-e37c-448d-a70d-96b5a218ac5e-credential-socket\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw\" (UID: \"823db35e-e37c-448d-a70d-96b5a218ac5e\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" Oct 31 01:12:07.276079 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.276058 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qmrf\" (UniqueName: \"kubernetes.io/projected/823db35e-e37c-448d-a70d-96b5a218ac5e-kube-api-access-6qmrf\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw\" (UID: \"823db35e-e37c-448d-a70d-96b5a218ac5e\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" Oct 31 01:12:07.276157 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.276094 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zmhr\" (UniqueName: \"kubernetes.io/projected/583a1183-7c84-40ce-aecb-9663f82a8a47-kube-api-access-8zmhr\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts\" (UID: \"583a1183-7c84-40ce-aecb-9663f82a8a47\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" Oct 31 01:12:07.276157 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.276136 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/583a1183-7c84-40ce-aecb-9663f82a8a47-workload-certs\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts\" (UID: \"583a1183-7c84-40ce-aecb-9663f82a8a47\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" Oct 31 01:12:07.276278 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.276162 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/583a1183-7c84-40ce-aecb-9663f82a8a47-istiod-ca-cert\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts\" (UID: \"583a1183-7c84-40ce-aecb-9663f82a8a47\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" Oct 31 01:12:07.276278 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.276205 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/823db35e-e37c-448d-a70d-96b5a218ac5e-istio-podinfo\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw\" (UID: \"823db35e-e37c-448d-a70d-96b5a218ac5e\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" Oct 31 01:12:07.276278 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.276234 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/583a1183-7c84-40ce-aecb-9663f82a8a47-istio-podinfo\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts\" (UID: \"583a1183-7c84-40ce-aecb-9663f82a8a47\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" Oct 31 01:12:07.276462 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.276297 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/823db35e-e37c-448d-a70d-96b5a218ac5e-istiod-ca-cert\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw\" (UID: \"823db35e-e37c-448d-a70d-96b5a218ac5e\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" Oct 31 01:12:07.276462 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.276374 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/583a1183-7c84-40ce-aecb-9663f82a8a47-credential-socket\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts\" (UID: \"583a1183-7c84-40ce-aecb-9663f82a8a47\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" Oct 31 01:12:07.276462 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.276408 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/823db35e-e37c-448d-a70d-96b5a218ac5e-istio-data\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw\" (UID: \"823db35e-e37c-448d-a70d-96b5a218ac5e\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" Oct 31 01:12:07.276462 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.276448 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/583a1183-7c84-40ce-aecb-9663f82a8a47-istio-token\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts\" (UID: \"583a1183-7c84-40ce-aecb-9663f82a8a47\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" Oct 31 01:12:07.276667 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.276575 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/823db35e-e37c-448d-a70d-96b5a218ac5e-istio-envoy\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw\" (UID: \"823db35e-e37c-448d-a70d-96b5a218ac5e\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" Oct 31 01:12:07.276667 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.276606 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/583a1183-7c84-40ce-aecb-9663f82a8a47-istio-data\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts\" (UID: \"583a1183-7c84-40ce-aecb-9663f82a8a47\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" Oct 31 01:12:07.276667 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.276635 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/823db35e-e37c-448d-a70d-96b5a218ac5e-workload-socket\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw\" (UID: \"823db35e-e37c-448d-a70d-96b5a218ac5e\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" Oct 31 01:12:07.276823 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.276672 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/823db35e-e37c-448d-a70d-96b5a218ac5e-workload-certs\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw\" (UID: \"823db35e-e37c-448d-a70d-96b5a218ac5e\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" Oct 31 01:12:07.276823 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.276697 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/823db35e-e37c-448d-a70d-96b5a218ac5e-istio-token\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw\" (UID: \"823db35e-e37c-448d-a70d-96b5a218ac5e\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" Oct 31 01:12:07.276823 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.276754 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/583a1183-7c84-40ce-aecb-9663f82a8a47-workload-socket\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts\" (UID: \"583a1183-7c84-40ce-aecb-9663f82a8a47\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" Oct 31 01:12:07.276823 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.276775 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/583a1183-7c84-40ce-aecb-9663f82a8a47-istio-envoy\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts\" (UID: \"583a1183-7c84-40ce-aecb-9663f82a8a47\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" Oct 31 01:12:07.277399 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.277242 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/583a1183-7c84-40ce-aecb-9663f82a8a47-credential-socket\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts\" (UID: \"583a1183-7c84-40ce-aecb-9663f82a8a47\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" Oct 31 01:12:07.285019 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.278100 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/583a1183-7c84-40ce-aecb-9663f82a8a47-istiod-ca-cert\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts\" (UID: \"583a1183-7c84-40ce-aecb-9663f82a8a47\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" Oct 31 01:12:07.285019 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.278577 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/583a1183-7c84-40ce-aecb-9663f82a8a47-workload-certs\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts\" (UID: \"583a1183-7c84-40ce-aecb-9663f82a8a47\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" Oct 31 01:12:07.285019 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.278894 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/583a1183-7c84-40ce-aecb-9663f82a8a47-istio-data\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts\" (UID: \"583a1183-7c84-40ce-aecb-9663f82a8a47\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" Oct 31 01:12:07.285019 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.279169 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/583a1183-7c84-40ce-aecb-9663f82a8a47-workload-socket\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts\" (UID: \"583a1183-7c84-40ce-aecb-9663f82a8a47\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" Oct 31 01:12:07.285019 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.280019 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/823db35e-e37c-448d-a70d-96b5a218ac5e-credential-socket\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw\" (UID: \"823db35e-e37c-448d-a70d-96b5a218ac5e\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" Oct 31 01:12:07.285019 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.280828 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/823db35e-e37c-448d-a70d-96b5a218ac5e-workload-certs\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw\" (UID: \"823db35e-e37c-448d-a70d-96b5a218ac5e\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" Oct 31 01:12:07.285019 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.281470 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/823db35e-e37c-448d-a70d-96b5a218ac5e-istiod-ca-cert\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw\" (UID: \"823db35e-e37c-448d-a70d-96b5a218ac5e\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" Oct 31 01:12:07.285019 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.281706 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/823db35e-e37c-448d-a70d-96b5a218ac5e-workload-socket\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw\" (UID: \"823db35e-e37c-448d-a70d-96b5a218ac5e\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" Oct 31 01:12:07.285019 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.281903 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/823db35e-e37c-448d-a70d-96b5a218ac5e-istio-data\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw\" (UID: \"823db35e-e37c-448d-a70d-96b5a218ac5e\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" Oct 31 01:12:07.285019 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.284532 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/823db35e-e37c-448d-a70d-96b5a218ac5e-istio-podinfo\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw\" (UID: \"823db35e-e37c-448d-a70d-96b5a218ac5e\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" Oct 31 01:12:07.285019 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.284945 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq"] Oct 31 01:12:07.286026 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.286003 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/583a1183-7c84-40ce-aecb-9663f82a8a47-istio-envoy\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts\" (UID: \"583a1183-7c84-40ce-aecb-9663f82a8a47\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" Oct 31 01:12:07.286294 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.286207 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/823db35e-e37c-448d-a70d-96b5a218ac5e-istio-envoy\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw\" (UID: \"823db35e-e37c-448d-a70d-96b5a218ac5e\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" Oct 31 01:12:07.286981 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.286960 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/583a1183-7c84-40ce-aecb-9663f82a8a47-istio-podinfo\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts\" (UID: \"583a1183-7c84-40ce-aecb-9663f82a8a47\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" Oct 31 01:12:07.291974 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.291947 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:12:07.292071 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.292007 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:12:07.292071 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.292037 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:12:07.328444 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.327286 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qmrf\" (UniqueName: \"kubernetes.io/projected/823db35e-e37c-448d-a70d-96b5a218ac5e-kube-api-access-6qmrf\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw\" (UID: \"823db35e-e37c-448d-a70d-96b5a218ac5e\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" Oct 31 01:12:07.329614 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.329543 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/583a1183-7c84-40ce-aecb-9663f82a8a47-istio-token\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts\" (UID: \"583a1183-7c84-40ce-aecb-9663f82a8a47\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" Oct 31 01:12:07.352632 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.339303 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zmhr\" (UniqueName: \"kubernetes.io/projected/583a1183-7c84-40ce-aecb-9663f82a8a47-kube-api-access-8zmhr\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts\" (UID: \"583a1183-7c84-40ce-aecb-9663f82a8a47\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" Oct 31 01:12:07.352632 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.340166 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/823db35e-e37c-448d-a70d-96b5a218ac5e-istio-token\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw\" (UID: \"823db35e-e37c-448d-a70d-96b5a218ac5e\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" Oct 31 01:12:07.352632 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.343679 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:07.363731 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.363702 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" Oct 31 01:12:07.390052 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.390028 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" Oct 31 01:12:07.592168 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:12:07.592127 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod613c5424_a9e2_43da_95aa_807e2b99ab2e.slice/crio-1167ebec6c98fdfd50951f4bbd10f8d8cad1cbcca7b18331dad032d916bd4bba WatchSource:0}: Error finding container 1167ebec6c98fdfd50951f4bbd10f8d8cad1cbcca7b18331dad032d916bd4bba: Status 404 returned error can't find the container with id 1167ebec6c98fdfd50951f4bbd10f8d8cad1cbcca7b18331dad032d916bd4bba Oct 31 01:12:07.592884 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.592765 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw"] Oct 31 01:12:07.594741 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.594707 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:12:07.594850 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.594784 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:12:07.594850 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.594824 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:12:07.678114 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:12:07.678084 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod583a1183_7c84_40ce_aecb_9663f82a8a47.slice/crio-32ae1d149f889e62f5298f82948d6c6f7257f04f5a97a173fe68b1c53193ffd5 WatchSource:0}: Error finding container 32ae1d149f889e62f5298f82948d6c6f7257f04f5a97a173fe68b1c53193ffd5: Status 404 returned error can't find the container with id 32ae1d149f889e62f5298f82948d6c6f7257f04f5a97a173fe68b1c53193ffd5 Oct 31 01:12:07.680509 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.680483 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts"] Oct 31 01:12:07.680792 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.680762 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:12:07.680872 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.680832 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:12:07.680872 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.680866 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:12:07.749055 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.741795 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw"] Oct 31 01:12:07.751193 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.751153 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:12:07.751319 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.751242 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:12:07.751319 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.751291 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:12:07.859285 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.859249 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" event={"ID":"823db35e-e37c-448d-a70d-96b5a218ac5e","Type":"ContainerStarted","Data":"305f28bdc13501bf40eb78a29b4b962936182bc5ac8c415e90ded874637b45d5"} Oct 31 01:12:07.859824 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.859312 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" event={"ID":"823db35e-e37c-448d-a70d-96b5a218ac5e","Type":"ContainerStarted","Data":"a190e0bbcfb64b6718490c35f249d351c116975a2d4774c2994bb4fe8a93651a"} Oct 31 01:12:07.862770 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.862738 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" event={"ID":"583a1183-7c84-40ce-aecb-9663f82a8a47","Type":"ContainerStarted","Data":"b6de0db241e1dcd09d56cd8b6d36c57f1a8e0a80d13e136c1b7080cc5d4f5044"} Oct 31 01:12:07.862907 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.862794 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" event={"ID":"583a1183-7c84-40ce-aecb-9663f82a8a47","Type":"ContainerStarted","Data":"32ae1d149f889e62f5298f82948d6c6f7257f04f5a97a173fe68b1c53193ffd5"} Oct 31 01:12:07.868461 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.865482 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" event={"ID":"613c5424-a9e2-43da-95aa-807e2b99ab2e","Type":"ContainerStarted","Data":"babe783bf64843c77c73c95295c0ff7e57a2de42a16a3f1e9d96e5e2820cd290"} Oct 31 01:12:07.868461 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.865531 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" event={"ID":"613c5424-a9e2-43da-95aa-807e2b99ab2e","Type":"ContainerStarted","Data":"1167ebec6c98fdfd50951f4bbd10f8d8cad1cbcca7b18331dad032d916bd4bba"} Oct 31 01:12:07.868949 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.868924 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" event={"ID":"4eb48867-402c-40ca-8db3-c513636cbf20","Type":"ContainerStarted","Data":"524b582fa2d49e50204960dff4f1821661b4b193b9f2f4aec0a23d8183a3764b"} Oct 31 01:12:07.869028 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.868972 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" event={"ID":"4eb48867-402c-40ca-8db3-c513636cbf20","Type":"ContainerStarted","Data":"5389c5e1689ac82c5762c8f01eff6169a7b1e6cc64a7432df01059bff28f8669"} Oct 31 01:12:07.887436 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.883775 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" podStartSLOduration=0.883758505 podStartE2EDuration="883.758505ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:12:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:07.882190016 +0000 UTC m=+972.240912331" watchObservedRunningTime="2026-10-31 01:12:07.883758505 +0000 UTC m=+972.242480893" Oct 31 01:12:07.951444 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.946581 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" Oct 31 01:12:07.990442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:07.990358 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" podStartSLOduration=1.990334016 podStartE2EDuration="1.990334016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:12:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:07.942477121 +0000 UTC m=+972.301199441" watchObservedRunningTime="2026-10-31 01:12:07.990334016 +0000 UTC m=+972.349056349" Oct 31 01:12:08.045266 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:08.045139 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" podStartSLOduration=2.045114979 podStartE2EDuration="2.045114979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:12:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:08.000635963 +0000 UTC m=+972.359358283" watchObservedRunningTime="2026-10-31 01:12:08.045114979 +0000 UTC m=+972.403837299" Oct 31 01:12:08.093233 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:08.093197 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:08.258012 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:08.257985 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" Oct 31 01:12:08.259394 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:08.259366 4440 patch_prober.go:29] interesting pod/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 01:12:08.259553 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:08.259446 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" podUID="613c5424-a9e2-43da-95aa-807e2b99ab2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 01:12:08.364757 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:08.364664 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" Oct 31 01:12:08.366226 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:08.366195 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 01:12:08.366363 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:08.366246 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" podUID="583a1183-7c84-40ce-aecb-9663f82a8a47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 01:12:08.390277 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:08.390250 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" Oct 31 01:12:08.391883 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:08.391858 4440 patch_prober.go:29] interesting pod/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 01:12:08.391998 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:08.391904 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" podUID="823db35e-e37c-448d-a70d-96b5a218ac5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 01:12:08.842306 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:08.842261 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:08.957828 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:08.957792 4440 patch_prober.go:29] interesting pod/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 01:12:08.958374 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:08.958343 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" podUID="4eb48867-402c-40ca-8db3-c513636cbf20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 01:12:09.267245 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:09.261652 4440 patch_prober.go:29] interesting pod/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 01:12:09.267245 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:09.261712 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" podUID="613c5424-a9e2-43da-95aa-807e2b99ab2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 01:12:09.365921 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:09.365778 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 01:12:09.365921 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:09.365843 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" podUID="583a1183-7c84-40ce-aecb-9663f82a8a47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 01:12:09.394443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:09.393661 4440 patch_prober.go:29] interesting pod/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 01:12:09.394443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:09.393730 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" podUID="823db35e-e37c-448d-a70d-96b5a218ac5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 01:12:09.789482 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:09.772614 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:09.821439 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:09.815729 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:09.950852 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:09.949777 4440 patch_prober.go:29] interesting pod/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 01:12:09.950852 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:09.949836 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" podUID="4eb48867-402c-40ca-8db3-c513636cbf20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 01:12:10.074527 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:10.074402 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:10.128178 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:10.128127 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" Oct 31 01:12:10.270444 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:10.269593 4440 patch_prober.go:29] interesting pod/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 01:12:10.270444 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:10.269649 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" podUID="613c5424-a9e2-43da-95aa-807e2b99ab2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 01:12:10.386503 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:10.381831 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 01:12:10.386503 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:10.381890 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" podUID="583a1183-7c84-40ce-aecb-9663f82a8a47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 01:12:10.390720 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:10.390689 4440 patch_prober.go:29] interesting pod/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 01:12:10.390946 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:10.390921 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" podUID="823db35e-e37c-448d-a70d-96b5a218ac5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 01:12:10.759221 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:10.759187 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" Oct 31 01:12:10.801233 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:10.801189 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:10.802079 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:10.802057 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" Oct 31 01:12:10.804432 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:10.804365 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" podStartSLOduration=4.804347447 podStartE2EDuration="4.804347447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:12:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:08.046469995 +0000 UTC m=+972.405192313" watchObservedRunningTime="2026-10-31 01:12:10.804347447 +0000 UTC m=+975.163069765" Oct 31 01:12:11.089499 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:11.073304 4440 patch_prober.go:29] interesting pod/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 01:12:11.089499 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:11.073362 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" podUID="4eb48867-402c-40ca-8db3-c513636cbf20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 01:12:11.103451 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:11.099599 4440 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:11.260201 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:11.260164 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:11.271502 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:11.271467 4440 patch_prober.go:29] interesting pod/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 01:12:11.271760 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:11.271730 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" podUID="613c5424-a9e2-43da-95aa-807e2b99ab2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 01:12:11.368792 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:11.367110 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 01:12:11.368792 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:11.367173 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" podUID="583a1183-7c84-40ce-aecb-9663f82a8a47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 01:12:11.387042 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:11.387004 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" Oct 31 01:12:11.390794 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:11.390766 4440 patch_prober.go:29] interesting pod/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 01:12:11.401560 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:11.401524 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" podUID="823db35e-e37c-448d-a70d-96b5a218ac5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 01:12:11.695533 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:11.645539 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:11.968370 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:11.968277 4440 patch_prober.go:29] interesting pod/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 01:12:11.976800 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:11.976760 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" podUID="4eb48867-402c-40ca-8db3-c513636cbf20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 01:12:12.201580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:12.201528 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:12.362857 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:12.341440 4440 patch_prober.go:29] interesting pod/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 01:12:12.366537 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:12.366501 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 01:12:12.366695 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:12.366559 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" podUID="583a1183-7c84-40ce-aecb-9663f82a8a47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 01:12:12.366695 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:12.366503 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" podUID="613c5424-a9e2-43da-95aa-807e2b99ab2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 01:12:12.401407 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:12.401358 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 31 01:12:12.419910 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:12.401991 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:12.420433 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:12.412760 4440 patch_prober.go:29] interesting pod/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 01:12:12.420614 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:12.420584 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" podUID="823db35e-e37c-448d-a70d-96b5a218ac5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 01:12:12.485443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:12.476915 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:12.812442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:12.811889 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" Oct 31 01:12:12.886442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:12.882086 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:13.071538 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:13.051799 4440 patch_prober.go:29] interesting pod/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 01:12:13.071538 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:13.051865 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" podUID="4eb48867-402c-40ca-8db3-c513636cbf20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 01:12:13.260442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:13.259832 4440 patch_prober.go:29] interesting pod/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 01:12:13.260442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:13.259912 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" podUID="613c5424-a9e2-43da-95aa-807e2b99ab2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 01:12:13.342504 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:13.337457 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:13.374443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:13.371051 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 01:12:13.374443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:13.371120 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" podUID="583a1183-7c84-40ce-aecb-9663f82a8a47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 01:12:13.397441 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:13.396520 4440 patch_prober.go:29] interesting pod/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 01:12:13.397441 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:13.396582 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" podUID="823db35e-e37c-448d-a70d-96b5a218ac5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 01:12:13.422442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:13.420496 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 31 01:12:13.422442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:13.421163 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" Oct 31 01:12:13.451985 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:13.451935 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:13.657515 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:13.653513 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:13.810440 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:13.803206 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:13.946812 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:13.946772 4440 patch_prober.go:29] interesting pod/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 01:12:13.947105 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:13.947075 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" podUID="4eb48867-402c-40ca-8db3-c513636cbf20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 01:12:14.101583 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:14.101536 4440 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:14.267516 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:14.266506 4440 patch_prober.go:29] interesting pod/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 01:12:14.267516 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:14.266571 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" podUID="613c5424-a9e2-43da-95aa-807e2b99ab2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 01:12:14.346242 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:14.346201 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 31 01:12:14.347081 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:14.347055 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" Oct 31 01:12:14.375442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:14.373449 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 01:12:14.375442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:14.373516 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" podUID="583a1183-7c84-40ce-aecb-9663f82a8a47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 01:12:14.375442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:14.373887 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:14.379768 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:14.379734 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:14.388553 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:14.388518 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 31 01:12:14.403157 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:14.403126 4440 patch_prober.go:29] interesting pod/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 01:12:14.403407 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:14.403385 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" podUID="823db35e-e37c-448d-a70d-96b5a218ac5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 01:12:14.615492 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:14.611659 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:14.615492 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:14.612011 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 01:12:14.615492 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:14.612750 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" Oct 31 01:12:14.677441 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:14.667219 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 01:12:14.754444 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:14.729017 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 31 01:12:14.811440 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:14.755062 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:14.811440 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:14.755598 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" Oct 31 01:12:14.858453 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:14.818546 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:14.858453 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:14.819239 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" Oct 31 01:12:14.959440 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:14.946448 4440 patch_prober.go:29] interesting pod/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 01:12:14.959440 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:14.946508 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" podUID="4eb48867-402c-40ca-8db3-c513636cbf20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 01:12:15.278142 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:15.278051 4440 patch_prober.go:29] interesting pod/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 01:12:15.278753 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:15.278721 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" podUID="613c5424-a9e2-43da-95aa-807e2b99ab2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 01:12:15.347671 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:15.347628 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 31 01:12:15.393910 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:15.390702 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 31 01:12:15.393910 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:15.391146 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" Oct 31 01:12:15.396029 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:15.394277 4440 patch_prober.go:29] interesting pod/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 01:12:15.396029 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:15.394321 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" podUID="823db35e-e37c-448d-a70d-96b5a218ac5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 01:12:15.402263 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:15.402234 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 01:12:15.402527 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:15.402484 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" podUID="583a1183-7c84-40ce-aecb-9663f82a8a47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 01:12:15.420767 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:15.420740 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" Oct 31 01:12:15.612345 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:15.612243 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:15.639758 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:15.612605 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 31 01:12:15.682443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:15.677960 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:15.682443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:15.678730 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" Oct 31 01:12:15.743057 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:15.743021 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:15.838774 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:15.838725 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:16.256067 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:16.256020 4440 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:16.265269 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:16.264651 4440 patch_prober.go:29] interesting pod/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 01:12:16.265269 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:16.264718 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" podUID="4eb48867-402c-40ca-8db3-c513636cbf20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 01:12:16.279547 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:16.278665 4440 patch_prober.go:29] interesting pod/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 01:12:16.279547 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:16.278723 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" podUID="613c5424-a9e2-43da-95aa-807e2b99ab2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 01:12:16.350707 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:16.350665 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:16.376217 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:16.370691 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 01:12:16.376217 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:16.370758 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" podUID="583a1183-7c84-40ce-aecb-9663f82a8a47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 01:12:16.390503 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:16.390462 4440 patch_prober.go:29] interesting pod/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 01:12:16.390688 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:16.390527 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" podUID="823db35e-e37c-448d-a70d-96b5a218ac5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 01:12:16.390688 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:16.390617 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:16.390955 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:16.390917 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:16.420600 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:16.420529 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:16.425132 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:16.425108 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" Oct 31 01:12:16.598838 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:16.598751 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" Oct 31 01:12:16.646839 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:16.646802 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" Oct 31 01:12:16.648568 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:16.648534 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:16.659392 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:16.652531 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" Oct 31 01:12:16.659724 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:16.659650 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:16.663734 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:16.663717 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" Oct 31 01:12:16.687476 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:16.687453 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" Oct 31 01:12:16.947900 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:16.947865 4440 patch_prober.go:29] interesting pod/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 01:12:16.950994 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:16.950962 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" podUID="4eb48867-402c-40ca-8db3-c513636cbf20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 01:12:17.263847 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:17.263764 4440 patch_prober.go:29] interesting pod/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 01:12:17.264124 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:17.264098 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" podUID="613c5424-a9e2-43da-95aa-807e2b99ab2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 01:12:17.404442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:17.385561 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 01:12:17.404442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:17.385624 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" podUID="583a1183-7c84-40ce-aecb-9663f82a8a47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 01:12:17.474441 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:17.445143 4440 patch_prober.go:29] interesting pod/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 01:12:17.474730 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:17.474690 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" podUID="823db35e-e37c-448d-a70d-96b5a218ac5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 01:12:18.072450 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:18.071569 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:18.448560 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:18.448496 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 31 01:12:18.449395 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:18.448634 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:18.450103 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:18.448680 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:18.522211 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:18.521932 4440 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:18.650569 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:18.650516 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:18.678663 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:18.676516 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:18.678663 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:18.678319 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:18.869497 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:18.849520 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:18.897699 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:18.897669 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" Oct 31 01:12:18.898954 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:18.898822 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" Oct 31 01:12:18.913044 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:18.911444 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" Oct 31 01:12:18.967494 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:18.965155 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:19.150694 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:19.150491 4440 patch_prober.go:29] interesting pod/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:12:19.150936 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:19.150905 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" podUID="cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:19.158882 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:19.158847 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:19.265637 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:19.265562 4440 patch_prober.go:29] interesting pod/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:12:19.265943 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:19.265894 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" podUID="613c5424-a9e2-43da-95aa-807e2b99ab2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:19.366406 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:19.366360 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 31 01:12:19.423439 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:19.419616 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" Oct 31 01:12:19.427437 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:19.424582 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" Oct 31 01:12:19.434446 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:19.432868 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" Oct 31 01:12:19.490763 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:19.490729 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:12:19.491322 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:19.491289 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" podUID="583a1183-7c84-40ce-aecb-9663f82a8a47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:19.491792 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:19.491764 4440 patch_prober.go:29] interesting pod/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:12:19.491985 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:19.491963 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" podUID="823db35e-e37c-448d-a70d-96b5a218ac5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:19.498535 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:19.498502 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:19.511440 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:19.499058 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" Oct 31 01:12:19.561541 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:19.561500 4440 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:19.586881 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:19.586849 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" Oct 31 01:12:19.754821 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:19.754735 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" Oct 31 01:12:19.756145 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:19.756126 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" Oct 31 01:12:19.789447 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:19.788073 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" Oct 31 01:12:19.919676 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:19.919496 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:20.447651 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:20.447619 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" Oct 31 01:12:20.448665 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:20.448635 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" Oct 31 01:12:20.449881 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:20.449863 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" Oct 31 01:12:20.678770 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:20.678741 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:21.393291 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:21.393254 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:22.090197 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.090164 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:22.145268 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.145230 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx"] Oct 31 01:12:22.151204 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.151167 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" Oct 31 01:12:22.184456 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.184395 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx"] Oct 31 01:12:22.280398 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.280355 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b29f36f2-9389-4f3c-91cf-658c373b901c-istiod-ca-cert\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx\" (UID: \"b29f36f2-9389-4f3c-91cf-658c373b901c\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" Oct 31 01:12:22.280620 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.280412 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b29f36f2-9389-4f3c-91cf-658c373b901c-istio-data\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx\" (UID: \"b29f36f2-9389-4f3c-91cf-658c373b901c\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" Oct 31 01:12:22.280620 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.280552 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b29f36f2-9389-4f3c-91cf-658c373b901c-credential-socket\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx\" (UID: \"b29f36f2-9389-4f3c-91cf-658c373b901c\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" Oct 31 01:12:22.280620 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.280609 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5w4l\" (UniqueName: \"kubernetes.io/projected/b29f36f2-9389-4f3c-91cf-658c373b901c-kube-api-access-m5w4l\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx\" (UID: \"b29f36f2-9389-4f3c-91cf-658c373b901c\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" Oct 31 01:12:22.280792 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.280710 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b29f36f2-9389-4f3c-91cf-658c373b901c-workload-socket\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx\" (UID: \"b29f36f2-9389-4f3c-91cf-658c373b901c\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" Oct 31 01:12:22.280940 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.280904 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b29f36f2-9389-4f3c-91cf-658c373b901c-istio-envoy\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx\" (UID: \"b29f36f2-9389-4f3c-91cf-658c373b901c\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" Oct 31 01:12:22.281015 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.280957 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b29f36f2-9389-4f3c-91cf-658c373b901c-istio-podinfo\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx\" (UID: \"b29f36f2-9389-4f3c-91cf-658c373b901c\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" Oct 31 01:12:22.281072 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.281013 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b29f36f2-9389-4f3c-91cf-658c373b901c-istio-token\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx\" (UID: \"b29f36f2-9389-4f3c-91cf-658c373b901c\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" Oct 31 01:12:22.281072 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.281047 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b29f36f2-9389-4f3c-91cf-658c373b901c-workload-certs\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx\" (UID: \"b29f36f2-9389-4f3c-91cf-658c373b901c\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" Oct 31 01:12:22.284227 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.284198 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw"] Oct 31 01:12:22.288400 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.288381 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" Oct 31 01:12:22.304426 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.304386 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw"] Oct 31 01:12:22.382336 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.382256 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b29f36f2-9389-4f3c-91cf-658c373b901c-istio-envoy\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx\" (UID: \"b29f36f2-9389-4f3c-91cf-658c373b901c\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" Oct 31 01:12:22.382336 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.382299 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee3eb69-94de-447c-b9ec-194a0458f2e7-workload-certs\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw\" (UID: \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" Oct 31 01:12:22.382336 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.382320 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b29f36f2-9389-4f3c-91cf-658c373b901c-istio-podinfo\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx\" (UID: \"b29f36f2-9389-4f3c-91cf-658c373b901c\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" Oct 31 01:12:22.382336 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.382336 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee3eb69-94de-447c-b9ec-194a0458f2e7-istio-token\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw\" (UID: \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" Oct 31 01:12:22.382723 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.382485 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b29f36f2-9389-4f3c-91cf-658c373b901c-istio-token\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx\" (UID: \"b29f36f2-9389-4f3c-91cf-658c373b901c\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" Oct 31 01:12:22.382723 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.382540 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee3eb69-94de-447c-b9ec-194a0458f2e7-istio-envoy\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw\" (UID: \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" Oct 31 01:12:22.382723 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.382574 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b29f36f2-9389-4f3c-91cf-658c373b901c-workload-certs\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx\" (UID: \"b29f36f2-9389-4f3c-91cf-658c373b901c\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" Oct 31 01:12:22.382723 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.382598 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee3eb69-94de-447c-b9ec-194a0458f2e7-workload-socket\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw\" (UID: \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" Oct 31 01:12:22.382723 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.382683 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee3eb69-94de-447c-b9ec-194a0458f2e7-istio-podinfo\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw\" (UID: \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" Oct 31 01:12:22.382976 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.382776 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b29f36f2-9389-4f3c-91cf-658c373b901c-istiod-ca-cert\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx\" (UID: \"b29f36f2-9389-4f3c-91cf-658c373b901c\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" Oct 31 01:12:22.382976 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.382805 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhc6g\" (UniqueName: \"kubernetes.io/projected/7ee3eb69-94de-447c-b9ec-194a0458f2e7-kube-api-access-rhc6g\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw\" (UID: \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" Oct 31 01:12:22.382976 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.382839 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b29f36f2-9389-4f3c-91cf-658c373b901c-istio-data\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx\" (UID: \"b29f36f2-9389-4f3c-91cf-658c373b901c\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" Oct 31 01:12:22.382976 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.382937 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b29f36f2-9389-4f3c-91cf-658c373b901c-credential-socket\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx\" (UID: \"b29f36f2-9389-4f3c-91cf-658c373b901c\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" Oct 31 01:12:22.383178 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.382976 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee3eb69-94de-447c-b9ec-194a0458f2e7-istio-data\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw\" (UID: \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" Oct 31 01:12:22.383178 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.382995 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b29f36f2-9389-4f3c-91cf-658c373b901c-workload-certs\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx\" (UID: \"b29f36f2-9389-4f3c-91cf-658c373b901c\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" Oct 31 01:12:22.383178 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.383043 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee3eb69-94de-447c-b9ec-194a0458f2e7-istiod-ca-cert\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw\" (UID: \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" Oct 31 01:12:22.383178 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.383123 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5w4l\" (UniqueName: \"kubernetes.io/projected/b29f36f2-9389-4f3c-91cf-658c373b901c-kube-api-access-m5w4l\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx\" (UID: \"b29f36f2-9389-4f3c-91cf-658c373b901c\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" Oct 31 01:12:22.383379 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.383184 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b29f36f2-9389-4f3c-91cf-658c373b901c-workload-socket\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx\" (UID: \"b29f36f2-9389-4f3c-91cf-658c373b901c\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" Oct 31 01:12:22.383379 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.383208 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b29f36f2-9389-4f3c-91cf-658c373b901c-istio-data\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx\" (UID: \"b29f36f2-9389-4f3c-91cf-658c373b901c\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" Oct 31 01:12:22.383379 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.383247 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b29f36f2-9389-4f3c-91cf-658c373b901c-credential-socket\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx\" (UID: \"b29f36f2-9389-4f3c-91cf-658c373b901c\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" Oct 31 01:12:22.383379 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.383306 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee3eb69-94de-447c-b9ec-194a0458f2e7-credential-socket\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw\" (UID: \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" Oct 31 01:12:22.383603 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.383529 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b29f36f2-9389-4f3c-91cf-658c373b901c-workload-socket\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx\" (UID: \"b29f36f2-9389-4f3c-91cf-658c373b901c\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" Oct 31 01:12:22.383954 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.383928 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b29f36f2-9389-4f3c-91cf-658c373b901c-istiod-ca-cert\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx\" (UID: \"b29f36f2-9389-4f3c-91cf-658c373b901c\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" Oct 31 01:12:22.384715 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.384691 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b29f36f2-9389-4f3c-91cf-658c373b901c-istio-envoy\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx\" (UID: \"b29f36f2-9389-4f3c-91cf-658c373b901c\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" Oct 31 01:12:22.385172 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.385152 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b29f36f2-9389-4f3c-91cf-658c373b901c-istio-podinfo\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx\" (UID: \"b29f36f2-9389-4f3c-91cf-658c373b901c\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" Oct 31 01:12:22.390993 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.390967 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b29f36f2-9389-4f3c-91cf-658c373b901c-istio-token\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx\" (UID: \"b29f36f2-9389-4f3c-91cf-658c373b901c\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" Oct 31 01:12:22.391186 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.391168 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5w4l\" (UniqueName: \"kubernetes.io/projected/b29f36f2-9389-4f3c-91cf-658c373b901c-kube-api-access-m5w4l\") pod \"e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx\" (UID: \"b29f36f2-9389-4f3c-91cf-658c373b901c\") " pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" Oct 31 01:12:22.484559 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.484532 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee3eb69-94de-447c-b9ec-194a0458f2e7-workload-certs\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw\" (UID: \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" Oct 31 01:12:22.484692 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.484565 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee3eb69-94de-447c-b9ec-194a0458f2e7-istio-token\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw\" (UID: \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" Oct 31 01:12:22.484692 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.484595 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee3eb69-94de-447c-b9ec-194a0458f2e7-istio-envoy\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw\" (UID: \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" Oct 31 01:12:22.484692 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.484614 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee3eb69-94de-447c-b9ec-194a0458f2e7-workload-socket\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw\" (UID: \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" Oct 31 01:12:22.484692 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.484641 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee3eb69-94de-447c-b9ec-194a0458f2e7-istio-podinfo\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw\" (UID: \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" Oct 31 01:12:22.484692 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.484667 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhc6g\" (UniqueName: \"kubernetes.io/projected/7ee3eb69-94de-447c-b9ec-194a0458f2e7-kube-api-access-rhc6g\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw\" (UID: \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" Oct 31 01:12:22.484970 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.484709 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee3eb69-94de-447c-b9ec-194a0458f2e7-istio-data\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw\" (UID: \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" Oct 31 01:12:22.484970 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.484758 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee3eb69-94de-447c-b9ec-194a0458f2e7-istiod-ca-cert\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw\" (UID: \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" Oct 31 01:12:22.484970 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.484881 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee3eb69-94de-447c-b9ec-194a0458f2e7-credential-socket\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw\" (UID: \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" Oct 31 01:12:22.485129 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.484982 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee3eb69-94de-447c-b9ec-194a0458f2e7-workload-certs\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw\" (UID: \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" Oct 31 01:12:22.485129 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.485102 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee3eb69-94de-447c-b9ec-194a0458f2e7-workload-socket\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw\" (UID: \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" Oct 31 01:12:22.485330 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.485310 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee3eb69-94de-447c-b9ec-194a0458f2e7-credential-socket\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw\" (UID: \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" Oct 31 01:12:22.485403 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.485365 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee3eb69-94de-447c-b9ec-194a0458f2e7-istio-data\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw\" (UID: \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" Oct 31 01:12:22.485572 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.485550 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee3eb69-94de-447c-b9ec-194a0458f2e7-istiod-ca-cert\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw\" (UID: \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" Oct 31 01:12:22.486840 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.486819 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee3eb69-94de-447c-b9ec-194a0458f2e7-istio-envoy\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw\" (UID: \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" Oct 31 01:12:22.487131 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.487113 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee3eb69-94de-447c-b9ec-194a0458f2e7-istio-podinfo\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw\" (UID: \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" Oct 31 01:12:22.491032 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.491004 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" Oct 31 01:12:22.493179 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.493158 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee3eb69-94de-447c-b9ec-194a0458f2e7-istio-token\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw\" (UID: \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" Oct 31 01:12:22.493397 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.493374 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhc6g\" (UniqueName: \"kubernetes.io/projected/7ee3eb69-94de-447c-b9ec-194a0458f2e7-kube-api-access-rhc6g\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw\" (UID: \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" Oct 31 01:12:22.608791 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.608760 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" Oct 31 01:12:22.787331 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.787300 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx"] Oct 31 01:12:22.787495 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:12:22.787472 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb29f36f2_9389_4f3c_91cf_658c373b901c.slice/crio-2fe8ddc03376c605c03ced5da69ed746eef52df0c301ea8bf2a6110addfc3ed1 WatchSource:0}: Error finding container 2fe8ddc03376c605c03ced5da69ed746eef52df0c301ea8bf2a6110addfc3ed1: Status 404 returned error can't find the container with id 2fe8ddc03376c605c03ced5da69ed746eef52df0c301ea8bf2a6110addfc3ed1 Oct 31 01:12:22.790127 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.790092 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:12:22.790215 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.790161 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:12:22.790215 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.790191 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:12:22.890485 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.890451 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:22.901857 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.901830 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw"] Oct 31 01:12:22.905347 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.905186 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:12:22.905347 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.905263 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:12:22.905347 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:22.905304 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:12:23.613441 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:23.613200 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" event={"ID":"7ee3eb69-94de-447c-b9ec-194a0458f2e7","Type":"ContainerStarted","Data":"6ced7779087cd90b78a891696327fcffe5daf642c5fe82ef100d9f7576911c49"} Oct 31 01:12:23.613441 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:23.613306 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" event={"ID":"7ee3eb69-94de-447c-b9ec-194a0458f2e7","Type":"ContainerStarted","Data":"76c9a5839359035a00ebea7b8bbbe1f5573ee1e3125d82987e0551ad2642c713"} Oct 31 01:12:23.634441 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:23.621604 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" event={"ID":"b29f36f2-9389-4f3c-91cf-658c373b901c","Type":"ContainerStarted","Data":"d077bbb89cc5bd53606420504b6da49534e3057fb292fea4d231c93ac785d681"} Oct 31 01:12:23.634441 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:23.621670 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" event={"ID":"b29f36f2-9389-4f3c-91cf-658c373b901c","Type":"ContainerStarted","Data":"2fe8ddc03376c605c03ced5da69ed746eef52df0c301ea8bf2a6110addfc3ed1"} Oct 31 01:12:23.675444 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:23.675215 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" podStartSLOduration=1.6751929749999999 podStartE2EDuration="1.675192975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:23.641733064 +0000 UTC m=+988.000455380" watchObservedRunningTime="2026-10-31 01:12:23.675192975 +0000 UTC m=+988.033915302" Oct 31 01:12:23.731748 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:23.731704 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:24.491467 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:24.491410 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" Oct 31 01:12:24.493146 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:24.493119 4440 patch_prober.go:29] interesting pod/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 31 01:12:24.493287 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:24.493182 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" podUID="b29f36f2-9389-4f3c-91cf-658c373b901c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 31 01:12:24.547439 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:24.546950 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:24.628486 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:24.617758 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" Oct 31 01:12:24.641489 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:24.629538 4440 patch_prober.go:29] interesting pod/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 01:12:24.641489 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:24.629599 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" podUID="7ee3eb69-94de-447c-b9ec-194a0458f2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 01:12:25.451443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:25.439436 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:25.491513 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:25.491478 4440 patch_prober.go:29] interesting pod/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 31 01:12:25.491771 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:25.491730 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" podUID="b29f36f2-9389-4f3c-91cf-658c373b901c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 31 01:12:25.618450 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:25.611779 4440 patch_prober.go:29] interesting pod/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 01:12:25.618450 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:25.611835 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" podUID="7ee3eb69-94de-447c-b9ec-194a0458f2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 01:12:26.492180 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:26.492144 4440 patch_prober.go:29] interesting pod/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 31 01:12:26.492565 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:26.492198 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" podUID="b29f36f2-9389-4f3c-91cf-658c373b901c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 31 01:12:26.557492 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:26.557461 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:26.622447 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:26.617837 4440 patch_prober.go:29] interesting pod/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 01:12:26.622447 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:26.617912 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" podUID="7ee3eb69-94de-447c-b9ec-194a0458f2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 01:12:26.776703 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:26.774226 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" podStartSLOduration=4.774208324 podStartE2EDuration="4.774208324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:23.672701222 +0000 UTC m=+988.031423537" watchObservedRunningTime="2026-10-31 01:12:26.774208324 +0000 UTC m=+991.132930639" Oct 31 01:12:26.776703 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:26.775204 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w"] Oct 31 01:12:26.779950 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:26.779927 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" Oct 31 01:12:26.784266 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:26.784244 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8499c8\"/\"openshift-service-ca.crt\"" Oct 31 01:12:26.788576 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:26.788524 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8499c8\"/\"default-dockercfg-qcw2m\"" Oct 31 01:12:26.788766 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:26.788748 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8499c8\"/\"kube-root-ca.crt\"" Oct 31 01:12:26.788976 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:26.788951 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8499c8\"/\"distinct-w4-8499c8-kserve-self-signed-certs\"" Oct 31 01:12:26.818482 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:26.796577 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w"] Oct 31 01:12:26.866238 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:26.854121 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b82b50e-bfa7-45b9-8484-c7a331b5ac03-tls-certs\") pod \"distinct-w4-8499c8-kserve-85599677b8-8tb2w\" (UID: \"0b82b50e-bfa7-45b9-8484-c7a331b5ac03\") " pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" Oct 31 01:12:26.866238 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:26.862823 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b82b50e-bfa7-45b9-8484-c7a331b5ac03-home\") pod \"distinct-w4-8499c8-kserve-85599677b8-8tb2w\" (UID: \"0b82b50e-bfa7-45b9-8484-c7a331b5ac03\") " pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" Oct 31 01:12:26.866238 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:26.862949 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b82b50e-bfa7-45b9-8484-c7a331b5ac03-dshm\") pod \"distinct-w4-8499c8-kserve-85599677b8-8tb2w\" (UID: \"0b82b50e-bfa7-45b9-8484-c7a331b5ac03\") " pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" Oct 31 01:12:26.866238 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:26.863024 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b82b50e-bfa7-45b9-8484-c7a331b5ac03-model-cache\") pod \"distinct-w4-8499c8-kserve-85599677b8-8tb2w\" (UID: \"0b82b50e-bfa7-45b9-8484-c7a331b5ac03\") " pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" Oct 31 01:12:26.866238 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:26.863207 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b82b50e-bfa7-45b9-8484-c7a331b5ac03-tmp-dir\") pod \"distinct-w4-8499c8-kserve-85599677b8-8tb2w\" (UID: \"0b82b50e-bfa7-45b9-8484-c7a331b5ac03\") " pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" Oct 31 01:12:26.866238 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:26.863239 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2qmr\" (UniqueName: \"kubernetes.io/projected/0b82b50e-bfa7-45b9-8484-c7a331b5ac03-kube-api-access-t2qmr\") pod \"distinct-w4-8499c8-kserve-85599677b8-8tb2w\" (UID: \"0b82b50e-bfa7-45b9-8484-c7a331b5ac03\") " pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" Oct 31 01:12:26.866685 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:26.866352 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:26.969215 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:26.968274 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b82b50e-bfa7-45b9-8484-c7a331b5ac03-model-cache\") pod \"distinct-w4-8499c8-kserve-85599677b8-8tb2w\" (UID: \"0b82b50e-bfa7-45b9-8484-c7a331b5ac03\") " pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" Oct 31 01:12:26.969215 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:26.968391 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b82b50e-bfa7-45b9-8484-c7a331b5ac03-tmp-dir\") pod \"distinct-w4-8499c8-kserve-85599677b8-8tb2w\" (UID: \"0b82b50e-bfa7-45b9-8484-c7a331b5ac03\") " pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" Oct 31 01:12:26.969215 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:26.968440 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2qmr\" (UniqueName: \"kubernetes.io/projected/0b82b50e-bfa7-45b9-8484-c7a331b5ac03-kube-api-access-t2qmr\") pod \"distinct-w4-8499c8-kserve-85599677b8-8tb2w\" (UID: \"0b82b50e-bfa7-45b9-8484-c7a331b5ac03\") " pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" Oct 31 01:12:26.969215 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:26.968578 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b82b50e-bfa7-45b9-8484-c7a331b5ac03-tls-certs\") pod \"distinct-w4-8499c8-kserve-85599677b8-8tb2w\" (UID: \"0b82b50e-bfa7-45b9-8484-c7a331b5ac03\") " pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" Oct 31 01:12:26.969215 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:26.968624 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b82b50e-bfa7-45b9-8484-c7a331b5ac03-home\") pod \"distinct-w4-8499c8-kserve-85599677b8-8tb2w\" (UID: \"0b82b50e-bfa7-45b9-8484-c7a331b5ac03\") " pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" Oct 31 01:12:26.969215 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:26.968731 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b82b50e-bfa7-45b9-8484-c7a331b5ac03-dshm\") pod \"distinct-w4-8499c8-kserve-85599677b8-8tb2w\" (UID: \"0b82b50e-bfa7-45b9-8484-c7a331b5ac03\") " pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" Oct 31 01:12:26.984606 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:26.974518 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b82b50e-bfa7-45b9-8484-c7a331b5ac03-model-cache\") pod \"distinct-w4-8499c8-kserve-85599677b8-8tb2w\" (UID: \"0b82b50e-bfa7-45b9-8484-c7a331b5ac03\") " pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" Oct 31 01:12:26.984864 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:26.974775 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b82b50e-bfa7-45b9-8484-c7a331b5ac03-home\") pod \"distinct-w4-8499c8-kserve-85599677b8-8tb2w\" (UID: \"0b82b50e-bfa7-45b9-8484-c7a331b5ac03\") " pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" Oct 31 01:12:26.984993 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:26.975019 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b82b50e-bfa7-45b9-8484-c7a331b5ac03-tmp-dir\") pod \"distinct-w4-8499c8-kserve-85599677b8-8tb2w\" (UID: \"0b82b50e-bfa7-45b9-8484-c7a331b5ac03\") " pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" Oct 31 01:12:27.002335 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:27.002265 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b82b50e-bfa7-45b9-8484-c7a331b5ac03-tls-certs\") pod \"distinct-w4-8499c8-kserve-85599677b8-8tb2w\" (UID: \"0b82b50e-bfa7-45b9-8484-c7a331b5ac03\") " pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" Oct 31 01:12:27.010444 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:27.008722 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b82b50e-bfa7-45b9-8484-c7a331b5ac03-dshm\") pod \"distinct-w4-8499c8-kserve-85599677b8-8tb2w\" (UID: \"0b82b50e-bfa7-45b9-8484-c7a331b5ac03\") " pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" Oct 31 01:12:27.010444 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:27.009078 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2qmr\" (UniqueName: \"kubernetes.io/projected/0b82b50e-bfa7-45b9-8484-c7a331b5ac03-kube-api-access-t2qmr\") pod \"distinct-w4-8499c8-kserve-85599677b8-8tb2w\" (UID: \"0b82b50e-bfa7-45b9-8484-c7a331b5ac03\") " pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" Oct 31 01:12:27.174357 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:27.174270 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" Oct 31 01:12:27.492837 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:27.492336 4440 patch_prober.go:29] interesting pod/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 31 01:12:27.492837 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:27.492398 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" podUID="b29f36f2-9389-4f3c-91cf-658c373b901c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 31 01:12:27.575458 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:12:27.575394 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b82b50e_bfa7_45b9_8484_c7a331b5ac03.slice/crio-b4882a9be9279714df783f4223f13f44a2c11556ef05b488c5731dc81346d4ab WatchSource:0}: Error finding container b4882a9be9279714df783f4223f13f44a2c11556ef05b488c5731dc81346d4ab: Status 404 returned error can't find the container with id b4882a9be9279714df783f4223f13f44a2c11556ef05b488c5731dc81346d4ab Oct 31 01:12:27.577012 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:27.576932 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w"] Oct 31 01:12:27.623443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:27.622536 4440 patch_prober.go:29] interesting pod/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 01:12:27.623443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:27.622596 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" podUID="7ee3eb69-94de-447c-b9ec-194a0458f2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 01:12:27.759383 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:27.758396 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:27.762458 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:27.762400 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" event={"ID":"0b82b50e-bfa7-45b9-8484-c7a331b5ac03","Type":"ContainerStarted","Data":"b4882a9be9279714df783f4223f13f44a2c11556ef05b488c5731dc81346d4ab"} Oct 31 01:12:28.491601 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:28.491569 4440 patch_prober.go:29] interesting pod/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 31 01:12:28.491879 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:28.491851 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" podUID="b29f36f2-9389-4f3c-91cf-658c373b901c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 31 01:12:28.615440 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:28.610758 4440 patch_prober.go:29] interesting pod/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 01:12:28.615440 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:28.610819 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" podUID="7ee3eb69-94de-447c-b9ec-194a0458f2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 01:12:28.649444 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:28.649380 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:28.790478 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:28.790388 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" event={"ID":"0b82b50e-bfa7-45b9-8484-c7a331b5ac03","Type":"ContainerStarted","Data":"d7b84ec0213b0cea20e0bb3b852fa1cecec5bbda6f4d8501bc5735874050bed8"} Oct 31 01:12:28.826530 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:28.826465 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" podStartSLOduration=2.8224488770000002 podStartE2EDuration="2.822448877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:28.819706 +0000 UTC m=+993.178428318" watchObservedRunningTime="2026-10-31 01:12:28.822448877 +0000 UTC m=+993.181171194" Oct 31 01:12:29.174825 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:29.174792 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" Oct 31 01:12:29.504815 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:29.504714 4440 patch_prober.go:29] interesting pod/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 31 01:12:29.504815 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:29.504794 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" podUID="b29f36f2-9389-4f3c-91cf-658c373b901c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 31 01:12:29.585655 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:29.585611 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:29.609390 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:29.609350 4440 patch_prober.go:29] interesting pod/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 01:12:29.609634 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:29.609481 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" podUID="7ee3eb69-94de-447c-b9ec-194a0458f2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 01:12:30.365815 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:30.365777 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:30.495511 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:30.495479 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" Oct 31 01:12:30.496027 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:30.496001 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" Oct 31 01:12:30.496801 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:30.496779 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" Oct 31 01:12:30.618536 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:30.618459 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" Oct 31 01:12:30.619480 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:30.618984 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" Oct 31 01:12:30.619875 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:30.619856 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" Oct 31 01:12:31.001822 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:12:31.001788 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:12:31.303381 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:31.303273 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:31.935704 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:31.935670 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:32.339147 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:32.339066 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4"] Oct 31 01:12:32.347594 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:32.347561 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4"] Oct 31 01:12:32.347757 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:32.347665 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" Oct 31 01:12:32.352449 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:32.352428 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-0a6cca\"/\"distinct-2-w2-0a6cca-kserve-self-signed-certs\"" Oct 31 01:12:32.391256 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:32.391209 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:32.514152 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:32.511873 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9c4h\" (UniqueName: \"kubernetes.io/projected/617e420d-d029-4f4b-8ae5-63fe7d85971e-kube-api-access-t9c4h\") pod \"distinct-2-w2-0a6cca-kserve-77868965c6-69cv4\" (UID: \"617e420d-d029-4f4b-8ae5-63fe7d85971e\") " pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" Oct 31 01:12:32.514152 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:32.511925 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/617e420d-d029-4f4b-8ae5-63fe7d85971e-dshm\") pod \"distinct-2-w2-0a6cca-kserve-77868965c6-69cv4\" (UID: \"617e420d-d029-4f4b-8ae5-63fe7d85971e\") " pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" Oct 31 01:12:32.514152 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:32.512148 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/617e420d-d029-4f4b-8ae5-63fe7d85971e-home\") pod \"distinct-2-w2-0a6cca-kserve-77868965c6-69cv4\" (UID: \"617e420d-d029-4f4b-8ae5-63fe7d85971e\") " pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" Oct 31 01:12:32.514152 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:32.512175 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/617e420d-d029-4f4b-8ae5-63fe7d85971e-model-cache\") pod \"distinct-2-w2-0a6cca-kserve-77868965c6-69cv4\" (UID: \"617e420d-d029-4f4b-8ae5-63fe7d85971e\") " pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" Oct 31 01:12:32.514152 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:32.512201 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/617e420d-d029-4f4b-8ae5-63fe7d85971e-tmp-dir\") pod \"distinct-2-w2-0a6cca-kserve-77868965c6-69cv4\" (UID: \"617e420d-d029-4f4b-8ae5-63fe7d85971e\") " pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" Oct 31 01:12:32.514152 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:32.512268 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/617e420d-d029-4f4b-8ae5-63fe7d85971e-tls-certs\") pod \"distinct-2-w2-0a6cca-kserve-77868965c6-69cv4\" (UID: \"617e420d-d029-4f4b-8ae5-63fe7d85971e\") " pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" Oct 31 01:12:32.613462 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:32.613336 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/617e420d-d029-4f4b-8ae5-63fe7d85971e-home\") pod \"distinct-2-w2-0a6cca-kserve-77868965c6-69cv4\" (UID: \"617e420d-d029-4f4b-8ae5-63fe7d85971e\") " pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" Oct 31 01:12:32.613462 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:32.613379 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/617e420d-d029-4f4b-8ae5-63fe7d85971e-model-cache\") pod \"distinct-2-w2-0a6cca-kserve-77868965c6-69cv4\" (UID: \"617e420d-d029-4f4b-8ae5-63fe7d85971e\") " pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" Oct 31 01:12:32.613462 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:32.613396 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/617e420d-d029-4f4b-8ae5-63fe7d85971e-tmp-dir\") pod \"distinct-2-w2-0a6cca-kserve-77868965c6-69cv4\" (UID: \"617e420d-d029-4f4b-8ae5-63fe7d85971e\") " pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" Oct 31 01:12:32.613757 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:32.613479 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/617e420d-d029-4f4b-8ae5-63fe7d85971e-tls-certs\") pod \"distinct-2-w2-0a6cca-kserve-77868965c6-69cv4\" (UID: \"617e420d-d029-4f4b-8ae5-63fe7d85971e\") " pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" Oct 31 01:12:32.613757 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:32.613656 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9c4h\" (UniqueName: \"kubernetes.io/projected/617e420d-d029-4f4b-8ae5-63fe7d85971e-kube-api-access-t9c4h\") pod \"distinct-2-w2-0a6cca-kserve-77868965c6-69cv4\" (UID: \"617e420d-d029-4f4b-8ae5-63fe7d85971e\") " pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" Oct 31 01:12:32.613757 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:32.613721 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/617e420d-d029-4f4b-8ae5-63fe7d85971e-dshm\") pod \"distinct-2-w2-0a6cca-kserve-77868965c6-69cv4\" (UID: \"617e420d-d029-4f4b-8ae5-63fe7d85971e\") " pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" Oct 31 01:12:32.613916 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:32.613821 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/617e420d-d029-4f4b-8ae5-63fe7d85971e-model-cache\") pod \"distinct-2-w2-0a6cca-kserve-77868965c6-69cv4\" (UID: \"617e420d-d029-4f4b-8ae5-63fe7d85971e\") " pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" Oct 31 01:12:32.613971 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:32.613928 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/617e420d-d029-4f4b-8ae5-63fe7d85971e-tmp-dir\") pod \"distinct-2-w2-0a6cca-kserve-77868965c6-69cv4\" (UID: \"617e420d-d029-4f4b-8ae5-63fe7d85971e\") " pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" Oct 31 01:12:32.613971 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:32.613929 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/617e420d-d029-4f4b-8ae5-63fe7d85971e-home\") pod \"distinct-2-w2-0a6cca-kserve-77868965c6-69cv4\" (UID: \"617e420d-d029-4f4b-8ae5-63fe7d85971e\") " pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" Oct 31 01:12:32.616762 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:32.616739 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/617e420d-d029-4f4b-8ae5-63fe7d85971e-dshm\") pod \"distinct-2-w2-0a6cca-kserve-77868965c6-69cv4\" (UID: \"617e420d-d029-4f4b-8ae5-63fe7d85971e\") " pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" Oct 31 01:12:32.617129 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:32.617107 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/617e420d-d029-4f4b-8ae5-63fe7d85971e-tls-certs\") pod \"distinct-2-w2-0a6cca-kserve-77868965c6-69cv4\" (UID: \"617e420d-d029-4f4b-8ae5-63fe7d85971e\") " pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" Oct 31 01:12:32.622100 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:32.622069 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9c4h\" (UniqueName: \"kubernetes.io/projected/617e420d-d029-4f4b-8ae5-63fe7d85971e-kube-api-access-t9c4h\") pod \"distinct-2-w2-0a6cca-kserve-77868965c6-69cv4\" (UID: \"617e420d-d029-4f4b-8ae5-63fe7d85971e\") " pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" Oct 31 01:12:32.665153 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:32.665126 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" Oct 31 01:12:32.840965 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:32.840927 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:32.956094 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:32.956066 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4"] Oct 31 01:12:32.956981 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:12:32.956954 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod617e420d_d029_4f4b_8ae5_63fe7d85971e.slice/crio-5ab066f043a74434317766b4e16a2aed8b5f7820ec60e46c36a2f261f101bb2d WatchSource:0}: Error finding container 5ab066f043a74434317766b4e16a2aed8b5f7820ec60e46c36a2f261f101bb2d: Status 404 returned error can't find the container with id 5ab066f043a74434317766b4e16a2aed8b5f7820ec60e46c36a2f261f101bb2d Oct 31 01:12:33.563398 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:33.563364 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:33.841781 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:33.841680 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" event={"ID":"617e420d-d029-4f4b-8ae5-63fe7d85971e","Type":"ContainerStarted","Data":"48f30cd271402c703cf948c7a6175c8b5b2778e52a098ec52cae25c96ee06d4b"} Oct 31 01:12:33.841781 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:33.841726 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" event={"ID":"617e420d-d029-4f4b-8ae5-63fe7d85971e","Type":"ContainerStarted","Data":"5ab066f043a74434317766b4e16a2aed8b5f7820ec60e46c36a2f261f101bb2d"} Oct 31 01:12:33.862226 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:33.862175 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" podStartSLOduration=1.862163029 podStartE2EDuration="1.862163029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:12:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:33.858839826 +0000 UTC m=+998.217562142" watchObservedRunningTime="2026-10-31 01:12:33.862163029 +0000 UTC m=+998.220885346" Oct 31 01:12:34.277104 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:34.277062 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:34.665331 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:34.665248 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" Oct 31 01:12:34.984666 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:34.984627 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:35.684527 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:35.684488 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:36.093137 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:36.093060 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:12:36.392623 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:36.392531 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:36.809087 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:36.809055 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4"] Oct 31 01:12:36.813906 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:36.813884 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" Oct 31 01:12:36.829468 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:36.829440 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4"] Oct 31 01:12:36.855900 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:36.855870 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph"] Oct 31 01:12:36.862088 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:36.862065 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" Oct 31 01:12:36.885853 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:36.885819 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph"] Oct 31 01:12:36.885998 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:36.885902 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4e5ddd-9933-47e1-9265-085858232375-workload-socket\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4\" (UID: \"3b4e5ddd-9933-47e1-9265-085858232375\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" Oct 31 01:12:36.886059 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:36.885994 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b4e5ddd-9933-47e1-9265-085858232375-istio-envoy\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4\" (UID: \"3b4e5ddd-9933-47e1-9265-085858232375\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" Oct 31 01:12:36.886110 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:36.886089 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4e5ddd-9933-47e1-9265-085858232375-credential-socket\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4\" (UID: \"3b4e5ddd-9933-47e1-9265-085858232375\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" Oct 31 01:12:36.886249 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:36.886227 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b4e5ddd-9933-47e1-9265-085858232375-istio-token\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4\" (UID: \"3b4e5ddd-9933-47e1-9265-085858232375\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" Oct 31 01:12:36.886329 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:36.886260 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b4e5ddd-9933-47e1-9265-085858232375-istiod-ca-cert\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4\" (UID: \"3b4e5ddd-9933-47e1-9265-085858232375\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" Oct 31 01:12:36.886329 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:36.886297 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnj6k\" (UniqueName: \"kubernetes.io/projected/3b4e5ddd-9933-47e1-9265-085858232375-kube-api-access-qnj6k\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4\" (UID: \"3b4e5ddd-9933-47e1-9265-085858232375\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" Oct 31 01:12:36.886468 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:36.886347 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b4e5ddd-9933-47e1-9265-085858232375-istio-podinfo\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4\" (UID: \"3b4e5ddd-9933-47e1-9265-085858232375\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" Oct 31 01:12:36.886525 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:36.886470 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b4e5ddd-9933-47e1-9265-085858232375-workload-certs\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4\" (UID: \"3b4e5ddd-9933-47e1-9265-085858232375\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" Oct 31 01:12:36.886525 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:36.886504 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b4e5ddd-9933-47e1-9265-085858232375-istio-data\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4\" (UID: \"3b4e5ddd-9933-47e1-9265-085858232375\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" Oct 31 01:12:37.012449 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.011716 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b4e5ddd-9933-47e1-9265-085858232375-workload-certs\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4\" (UID: \"3b4e5ddd-9933-47e1-9265-085858232375\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" Oct 31 01:12:37.012449 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.011795 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b4e5ddd-9933-47e1-9265-085858232375-istio-data\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4\" (UID: \"3b4e5ddd-9933-47e1-9265-085858232375\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" Oct 31 01:12:37.012449 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.011871 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4e5ddd-9933-47e1-9265-085858232375-workload-socket\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4\" (UID: \"3b4e5ddd-9933-47e1-9265-085858232375\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" Oct 31 01:12:37.012449 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.011934 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b192b955-7018-4d48-b3c6-c58d86eeb5b2-istio-podinfo\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph\" (UID: \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" Oct 31 01:12:37.012449 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.012007 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b4e5ddd-9933-47e1-9265-085858232375-istio-envoy\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4\" (UID: \"3b4e5ddd-9933-47e1-9265-085858232375\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" Oct 31 01:12:37.012449 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.012034 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpb6q\" (UniqueName: \"kubernetes.io/projected/b192b955-7018-4d48-b3c6-c58d86eeb5b2-kube-api-access-gpb6q\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph\" (UID: \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" Oct 31 01:12:37.012449 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.012080 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b192b955-7018-4d48-b3c6-c58d86eeb5b2-istiod-ca-cert\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph\" (UID: \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" Oct 31 01:12:37.012449 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.012141 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b192b955-7018-4d48-b3c6-c58d86eeb5b2-istio-envoy\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph\" (UID: \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" Oct 31 01:12:37.012449 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.012176 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b192b955-7018-4d48-b3c6-c58d86eeb5b2-workload-socket\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph\" (UID: \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" Oct 31 01:12:37.012449 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.012256 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b192b955-7018-4d48-b3c6-c58d86eeb5b2-istio-data\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph\" (UID: \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" Oct 31 01:12:37.012449 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.012370 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4e5ddd-9933-47e1-9265-085858232375-credential-socket\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4\" (UID: \"3b4e5ddd-9933-47e1-9265-085858232375\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" Oct 31 01:12:37.012449 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.012410 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b4e5ddd-9933-47e1-9265-085858232375-istio-token\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4\" (UID: \"3b4e5ddd-9933-47e1-9265-085858232375\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" Oct 31 01:12:37.026476 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.023773 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b4e5ddd-9933-47e1-9265-085858232375-istiod-ca-cert\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4\" (UID: \"3b4e5ddd-9933-47e1-9265-085858232375\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" Oct 31 01:12:37.026476 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.023876 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnj6k\" (UniqueName: \"kubernetes.io/projected/3b4e5ddd-9933-47e1-9265-085858232375-kube-api-access-qnj6k\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4\" (UID: \"3b4e5ddd-9933-47e1-9265-085858232375\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" Oct 31 01:12:37.026476 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.023967 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b4e5ddd-9933-47e1-9265-085858232375-istio-podinfo\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4\" (UID: \"3b4e5ddd-9933-47e1-9265-085858232375\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" Oct 31 01:12:37.026476 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.024015 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b192b955-7018-4d48-b3c6-c58d86eeb5b2-credential-socket\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph\" (UID: \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" Oct 31 01:12:37.026476 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.024069 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b192b955-7018-4d48-b3c6-c58d86eeb5b2-istio-token\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph\" (UID: \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" Oct 31 01:12:37.026476 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.024176 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b192b955-7018-4d48-b3c6-c58d86eeb5b2-workload-certs\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph\" (UID: \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" Oct 31 01:12:37.026476 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.016580 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b4e5ddd-9933-47e1-9265-085858232375-istio-data\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4\" (UID: \"3b4e5ddd-9933-47e1-9265-085858232375\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" Oct 31 01:12:37.026476 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.016298 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b4e5ddd-9933-47e1-9265-085858232375-workload-certs\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4\" (UID: \"3b4e5ddd-9933-47e1-9265-085858232375\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" Oct 31 01:12:37.026476 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.017937 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4e5ddd-9933-47e1-9265-085858232375-credential-socket\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4\" (UID: \"3b4e5ddd-9933-47e1-9265-085858232375\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" Oct 31 01:12:37.026476 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.016822 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4e5ddd-9933-47e1-9265-085858232375-workload-socket\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4\" (UID: \"3b4e5ddd-9933-47e1-9265-085858232375\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" Oct 31 01:12:37.028466 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.028443 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b4e5ddd-9933-47e1-9265-085858232375-istiod-ca-cert\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4\" (UID: \"3b4e5ddd-9933-47e1-9265-085858232375\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" Oct 31 01:12:37.041130 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.041097 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b4e5ddd-9933-47e1-9265-085858232375-istio-podinfo\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4\" (UID: \"3b4e5ddd-9933-47e1-9265-085858232375\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" Oct 31 01:12:37.041320 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.041294 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b4e5ddd-9933-47e1-9265-085858232375-istio-envoy\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4\" (UID: \"3b4e5ddd-9933-47e1-9265-085858232375\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" Oct 31 01:12:37.045349 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.044666 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnj6k\" (UniqueName: \"kubernetes.io/projected/3b4e5ddd-9933-47e1-9265-085858232375-kube-api-access-qnj6k\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4\" (UID: \"3b4e5ddd-9933-47e1-9265-085858232375\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" Oct 31 01:12:37.060528 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.056958 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b4e5ddd-9933-47e1-9265-085858232375-istio-token\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4\" (UID: \"3b4e5ddd-9933-47e1-9265-085858232375\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" Oct 31 01:12:37.125442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.125377 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b192b955-7018-4d48-b3c6-c58d86eeb5b2-istio-podinfo\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph\" (UID: \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" Oct 31 01:12:37.125652 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.125473 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpb6q\" (UniqueName: \"kubernetes.io/projected/b192b955-7018-4d48-b3c6-c58d86eeb5b2-kube-api-access-gpb6q\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph\" (UID: \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" Oct 31 01:12:37.125652 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.125504 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b192b955-7018-4d48-b3c6-c58d86eeb5b2-istiod-ca-cert\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph\" (UID: \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" Oct 31 01:12:37.125652 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.125545 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b192b955-7018-4d48-b3c6-c58d86eeb5b2-istio-envoy\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph\" (UID: \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" Oct 31 01:12:37.125652 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.125574 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b192b955-7018-4d48-b3c6-c58d86eeb5b2-workload-socket\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph\" (UID: \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" Oct 31 01:12:37.125652 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.125628 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b192b955-7018-4d48-b3c6-c58d86eeb5b2-istio-data\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph\" (UID: \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" Oct 31 01:12:37.125945 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.125783 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b192b955-7018-4d48-b3c6-c58d86eeb5b2-credential-socket\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph\" (UID: \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" Oct 31 01:12:37.125945 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.125829 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b192b955-7018-4d48-b3c6-c58d86eeb5b2-istio-token\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph\" (UID: \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" Oct 31 01:12:37.125945 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.125911 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b192b955-7018-4d48-b3c6-c58d86eeb5b2-workload-certs\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph\" (UID: \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" Oct 31 01:12:37.129437 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.126398 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b192b955-7018-4d48-b3c6-c58d86eeb5b2-workload-certs\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph\" (UID: \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" Oct 31 01:12:37.129437 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.128955 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b192b955-7018-4d48-b3c6-c58d86eeb5b2-workload-socket\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph\" (UID: \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" Oct 31 01:12:37.129437 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.129399 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b192b955-7018-4d48-b3c6-c58d86eeb5b2-istio-data\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph\" (UID: \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" Oct 31 01:12:37.139468 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.129692 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b192b955-7018-4d48-b3c6-c58d86eeb5b2-credential-socket\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph\" (UID: \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" Oct 31 01:12:37.139468 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.129945 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b192b955-7018-4d48-b3c6-c58d86eeb5b2-istiod-ca-cert\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph\" (UID: \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" Oct 31 01:12:37.139468 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.132017 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" Oct 31 01:12:37.139468 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.135113 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b192b955-7018-4d48-b3c6-c58d86eeb5b2-istio-envoy\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph\" (UID: \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" Oct 31 01:12:37.139468 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.135622 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b192b955-7018-4d48-b3c6-c58d86eeb5b2-istio-podinfo\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph\" (UID: \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" Oct 31 01:12:37.146524 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.145993 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpb6q\" (UniqueName: \"kubernetes.io/projected/b192b955-7018-4d48-b3c6-c58d86eeb5b2-kube-api-access-gpb6q\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph\" (UID: \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" Oct 31 01:12:37.146524 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.146408 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b192b955-7018-4d48-b3c6-c58d86eeb5b2-istio-token\") pod \"e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph\" (UID: \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\") " pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" Oct 31 01:12:37.175225 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.175186 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" Oct 31 01:12:37.193907 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.193881 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" Oct 31 01:12:37.200844 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.197616 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" Oct 31 01:12:37.539104 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.538815 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:37.707610 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.687116 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:37.762486 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.759124 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:12:37.762486 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.759226 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:12:37.762486 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:37.759273 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:12:38.104450 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:38.100488 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4"] Oct 31 01:12:38.281441 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:38.277838 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:12:38.281816 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:38.281776 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:12:38.282049 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:38.282021 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:12:38.360329 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:38.360244 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" event={"ID":"3b4e5ddd-9933-47e1-9265-085858232375","Type":"ContainerStarted","Data":"f5013138e7196fda65733f7931aade76b2865fa9ad1eab8a0e94266177487e3d"} Oct 31 01:12:38.421705 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:38.421671 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph"] Oct 31 01:12:38.574469 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:38.570534 4440 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:38.699553 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:38.699506 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 31 01:12:38.700163 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:38.700125 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 01:12:38.700635 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:38.700611 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 31 01:12:38.706435 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:38.702844 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:38.885411 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:38.884788 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 31 01:12:39.199456 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:39.191877 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" podUID="0b82b50e-bfa7-45b9-8484-c7a331b5ac03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:39.365439 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:39.362561 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:39.396604 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:39.396562 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:39.426847 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:39.426800 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:39.457600 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:39.457502 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" event={"ID":"3b4e5ddd-9933-47e1-9265-085858232375","Type":"ContainerStarted","Data":"9159b732b08f6092f3f7cf448af44053edff619b23624a91e1b5ae0f215011e0"} Oct 31 01:12:39.544446 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:39.533880 4440 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded" Oct 31 01:12:39.582489 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:39.582367 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" event={"ID":"b192b955-7018-4d48-b3c6-c58d86eeb5b2","Type":"ContainerStarted","Data":"8a419a62c34d493bacbfce57925ee0e68c8d090600939ab0e06825ba43becc17"} Oct 31 01:12:39.582787 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:39.582711 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" event={"ID":"b192b955-7018-4d48-b3c6-c58d86eeb5b2","Type":"ContainerStarted","Data":"bcab6fb945016f475c917bb990350de5fe1eb8e07f36ab701d4e351c79f3de18"} Oct 31 01:12:39.596443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:39.595025 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" podStartSLOduration=3.5949959849999997 podStartE2EDuration="3.594995985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:39.559001749 +0000 UTC m=+1003.917724067" watchObservedRunningTime="2026-10-31 01:12:39.594995985 +0000 UTC m=+1003.953718302" Oct 31 01:12:39.655452 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:39.652017 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:39.684702 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:39.684630 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" podStartSLOduration=3.684608517 podStartE2EDuration="3.684608517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:39.605041564 +0000 UTC m=+1003.963763916" watchObservedRunningTime="2026-10-31 01:12:39.684608517 +0000 UTC m=+1004.043330836" Oct 31 01:12:39.752491 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:39.717886 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:39.752491 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:39.718255 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 31 01:12:39.752491 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:39.718543 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 01:12:39.752491 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:39.719668 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 31 01:12:39.752491 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:39.720296 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" Oct 31 01:12:39.752491 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:39.720369 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" Oct 31 01:12:39.752491 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:39.720409 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" Oct 31 01:12:39.752491 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:39.720467 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" Oct 31 01:12:40.090387 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:40.089912 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:40.135439 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:40.134457 4440 patch_prober.go:29] interesting pod/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 01:12:40.135439 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:40.134517 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" podUID="3b4e5ddd-9933-47e1-9265-085858232375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 01:12:40.255631 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:40.192899 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" podUID="0b82b50e-bfa7-45b9-8484-c7a331b5ac03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:40.294470 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:40.256477 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:40.361019 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:40.360931 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" Oct 31 01:12:40.445115 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:40.445087 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" Oct 31 01:12:40.445413 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:40.445395 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878"] Oct 31 01:12:40.445836 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:40.433524 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:40.446278 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:40.365473 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:40.488759 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:40.432513 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:40.649501 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:40.635466 4440 patch_prober.go:29] interesting pod/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 31 01:12:40.649501 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:40.635527 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" podUID="b192b955-7018-4d48-b3c6-c58d86eeb5b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 31 01:12:40.779925 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:40.774319 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" Oct 31 01:12:40.785298 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:40.785269 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" Oct 31 01:12:40.785514 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:40.785321 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" Oct 31 01:12:40.785514 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:40.785355 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" Oct 31 01:12:40.785514 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:40.785395 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" Oct 31 01:12:40.785514 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:40.785430 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878"] Oct 31 01:12:40.785514 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:40.785471 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" Oct 31 01:12:40.785514 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:40.785504 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" Oct 31 01:12:40.786235 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:40.785534 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" Oct 31 01:12:40.786235 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:40.785574 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" Oct 31 01:12:40.786235 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:40.778570 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" Oct 31 01:12:40.793364 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:40.793336 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" Oct 31 01:12:40.813468 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:40.800524 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:12:40.813468 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:40.801822 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-28ff08\"/\"distinct-2-w5-28ff08-kserve-self-signed-certs\"" Oct 31 01:12:40.813468 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:40.805922 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" Oct 31 01:12:40.954835 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:40.954800 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf86081d-7942-454a-8763-f658bd0ce870-tls-certs\") pod \"distinct-2-w5-28ff08-kserve-7cbb68c869-wk878\" (UID: \"cf86081d-7942-454a-8763-f658bd0ce870\") " pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" Oct 31 01:12:40.955816 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:40.955776 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf86081d-7942-454a-8763-f658bd0ce870-dshm\") pod \"distinct-2-w5-28ff08-kserve-7cbb68c869-wk878\" (UID: \"cf86081d-7942-454a-8763-f658bd0ce870\") " pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" Oct 31 01:12:40.957554 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:40.957534 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf86081d-7942-454a-8763-f658bd0ce870-home\") pod \"distinct-2-w5-28ff08-kserve-7cbb68c869-wk878\" (UID: \"cf86081d-7942-454a-8763-f658bd0ce870\") " pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" Oct 31 01:12:40.958323 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:40.958306 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf86081d-7942-454a-8763-f658bd0ce870-tmp-dir\") pod \"distinct-2-w5-28ff08-kserve-7cbb68c869-wk878\" (UID: \"cf86081d-7942-454a-8763-f658bd0ce870\") " pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" Oct 31 01:12:40.959430 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:40.959398 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d2jj\" (UniqueName: \"kubernetes.io/projected/cf86081d-7942-454a-8763-f658bd0ce870-kube-api-access-5d2jj\") pod \"distinct-2-w5-28ff08-kserve-7cbb68c869-wk878\" (UID: \"cf86081d-7942-454a-8763-f658bd0ce870\") " pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" Oct 31 01:12:40.960296 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:40.960276 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf86081d-7942-454a-8763-f658bd0ce870-model-cache\") pod \"distinct-2-w5-28ff08-kserve-7cbb68c869-wk878\" (UID: \"cf86081d-7942-454a-8763-f658bd0ce870\") " pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" Oct 31 01:12:41.031442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:41.021007 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" Oct 31 01:12:41.036723 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:41.036695 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" Oct 31 01:12:41.070605 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:41.070539 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf86081d-7942-454a-8763-f658bd0ce870-model-cache\") pod \"distinct-2-w5-28ff08-kserve-7cbb68c869-wk878\" (UID: \"cf86081d-7942-454a-8763-f658bd0ce870\") " pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" Oct 31 01:12:41.070780 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:41.070679 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf86081d-7942-454a-8763-f658bd0ce870-tls-certs\") pod \"distinct-2-w5-28ff08-kserve-7cbb68c869-wk878\" (UID: \"cf86081d-7942-454a-8763-f658bd0ce870\") " pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" Oct 31 01:12:41.070882 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:41.070864 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf86081d-7942-454a-8763-f658bd0ce870-dshm\") pod \"distinct-2-w5-28ff08-kserve-7cbb68c869-wk878\" (UID: \"cf86081d-7942-454a-8763-f658bd0ce870\") " pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" Oct 31 01:12:41.070940 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:41.070917 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf86081d-7942-454a-8763-f658bd0ce870-home\") pod \"distinct-2-w5-28ff08-kserve-7cbb68c869-wk878\" (UID: \"cf86081d-7942-454a-8763-f658bd0ce870\") " pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" Oct 31 01:12:41.071002 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:41.070969 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf86081d-7942-454a-8763-f658bd0ce870-tmp-dir\") pod \"distinct-2-w5-28ff08-kserve-7cbb68c869-wk878\" (UID: \"cf86081d-7942-454a-8763-f658bd0ce870\") " pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" Oct 31 01:12:41.071311 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:41.071088 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d2jj\" (UniqueName: \"kubernetes.io/projected/cf86081d-7942-454a-8763-f658bd0ce870-kube-api-access-5d2jj\") pod \"distinct-2-w5-28ff08-kserve-7cbb68c869-wk878\" (UID: \"cf86081d-7942-454a-8763-f658bd0ce870\") " pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" Oct 31 01:12:41.102257 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:41.095488 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf86081d-7942-454a-8763-f658bd0ce870-home\") pod \"distinct-2-w5-28ff08-kserve-7cbb68c869-wk878\" (UID: \"cf86081d-7942-454a-8763-f658bd0ce870\") " pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" Oct 31 01:12:41.102257 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:41.095786 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf86081d-7942-454a-8763-f658bd0ce870-tmp-dir\") pod \"distinct-2-w5-28ff08-kserve-7cbb68c869-wk878\" (UID: \"cf86081d-7942-454a-8763-f658bd0ce870\") " pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" Oct 31 01:12:41.147746 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:41.147697 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf86081d-7942-454a-8763-f658bd0ce870-model-cache\") pod \"distinct-2-w5-28ff08-kserve-7cbb68c869-wk878\" (UID: \"cf86081d-7942-454a-8763-f658bd0ce870\") " pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" Oct 31 01:12:41.148640 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:41.148592 4440 patch_prober.go:29] interesting pod/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 01:12:41.148760 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:41.148667 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" podUID="3b4e5ddd-9933-47e1-9265-085858232375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 01:12:41.192834 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:41.192783 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d2jj\" (UniqueName: \"kubernetes.io/projected/cf86081d-7942-454a-8763-f658bd0ce870-kube-api-access-5d2jj\") pod \"distinct-2-w5-28ff08-kserve-7cbb68c869-wk878\" (UID: \"cf86081d-7942-454a-8763-f658bd0ce870\") " pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" Oct 31 01:12:41.195071 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:41.194960 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf86081d-7942-454a-8763-f658bd0ce870-dshm\") pod \"distinct-2-w5-28ff08-kserve-7cbb68c869-wk878\" (UID: \"cf86081d-7942-454a-8763-f658bd0ce870\") " pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" Oct 31 01:12:41.209558 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:41.205085 4440 patch_prober.go:29] interesting pod/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 31 01:12:41.209558 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:41.205143 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" podUID="b192b955-7018-4d48-b3c6-c58d86eeb5b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 31 01:12:41.254615 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:41.218095 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf86081d-7942-454a-8763-f658bd0ce870-tls-certs\") pod \"distinct-2-w5-28ff08-kserve-7cbb68c869-wk878\" (UID: \"cf86081d-7942-454a-8763-f658bd0ce870\") " pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" Oct 31 01:12:41.278667 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:41.278599 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:41.486324 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:41.485990 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" Oct 31 01:12:41.964523 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:41.964083 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878"] Oct 31 01:12:42.132835 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:42.132797 4440 patch_prober.go:29] interesting pod/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 01:12:42.133913 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:42.133880 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" podUID="3b4e5ddd-9933-47e1-9265-085858232375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 01:12:42.199821 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:42.199780 4440 patch_prober.go:29] interesting pod/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 31 01:12:42.200099 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:42.200070 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" podUID="b192b955-7018-4d48-b3c6-c58d86eeb5b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 31 01:12:42.258160 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:42.258065 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:42.665952 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:42.665847 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" Oct 31 01:12:42.678875 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:42.678845 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" Oct 31 01:12:42.893062 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:42.893027 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" event={"ID":"cf86081d-7942-454a-8763-f658bd0ce870","Type":"ContainerStarted","Data":"c4efb30afb1028e371b427a798ca232592bde14d3173fc46f372a61d911f535f"} Oct 31 01:12:42.893244 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:42.893072 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" event={"ID":"cf86081d-7942-454a-8763-f658bd0ce870","Type":"ContainerStarted","Data":"5424adcadc84a2efd5dbc04457af3ec2b7e4f99ef8bda69156289a22fd99716e"} Oct 31 01:12:42.935443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:42.929871 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" podStartSLOduration=3.929849092 podStartE2EDuration="3.929849092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:42.92382492 +0000 UTC m=+1007.282547237" watchObservedRunningTime="2026-10-31 01:12:42.929849092 +0000 UTC m=+1007.288571411" Oct 31 01:12:42.962215 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:42.962182 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" Oct 31 01:12:43.133860 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:43.133815 4440 patch_prober.go:29] interesting pod/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 01:12:43.157559 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:43.157510 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" podUID="3b4e5ddd-9933-47e1-9265-085858232375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 01:12:43.223498 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:43.214740 4440 patch_prober.go:29] interesting pod/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 31 01:12:43.223498 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:43.214819 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" podUID="b192b955-7018-4d48-b3c6-c58d86eeb5b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 31 01:12:43.514501 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:43.506500 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" Oct 31 01:12:43.669151 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:43.669115 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:43.781966 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:43.781869 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:44.138491 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:44.138336 4440 patch_prober.go:29] interesting pod/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 01:12:44.138491 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:44.138410 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" podUID="3b4e5ddd-9933-47e1-9265-085858232375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 01:12:44.203440 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:44.202836 4440 patch_prober.go:29] interesting pod/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 31 01:12:44.203440 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:44.202895 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" podUID="b192b955-7018-4d48-b3c6-c58d86eeb5b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 31 01:12:45.132853 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:45.132815 4440 patch_prober.go:29] interesting pod/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 01:12:45.133485 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:45.133451 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" podUID="3b4e5ddd-9933-47e1-9265-085858232375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 01:12:45.213443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:45.199735 4440 patch_prober.go:29] interesting pod/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 31 01:12:45.213443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:45.199796 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" podUID="b192b955-7018-4d48-b3c6-c58d86eeb5b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 31 01:12:45.467876 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:45.467664 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:45.953448 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:45.951307 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:46.134444 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:46.133773 4440 patch_prober.go:29] interesting pod/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 01:12:46.134444 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:46.133831 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" podUID="3b4e5ddd-9933-47e1-9265-085858232375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 01:12:46.208500 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:46.200806 4440 patch_prober.go:29] interesting pod/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 31 01:12:46.208500 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:46.200868 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" podUID="b192b955-7018-4d48-b3c6-c58d86eeb5b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 31 01:12:46.782449 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:46.782392 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:47.133610 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:47.133508 4440 patch_prober.go:29] interesting pod/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 01:12:47.133610 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:47.133585 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" podUID="3b4e5ddd-9933-47e1-9265-085858232375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 01:12:47.321761 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:47.321722 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" Oct 31 01:12:47.323809 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:47.323410 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" Oct 31 01:12:47.324897 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:47.324870 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" Oct 31 01:12:47.894616 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:47.894523 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:48.137769 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:48.137732 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" Oct 31 01:12:48.138178 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:48.138156 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" Oct 31 01:12:48.138834 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:48.138813 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" Oct 31 01:12:48.778862 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:48.778792 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:49.480851 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:49.480805 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:50.229358 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:50.229318 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:50.908386 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:50.908346 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:51.505463 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:51.505401 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" Oct 31 01:12:51.518768 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:51.518740 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" Oct 31 01:12:51.646063 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:51.646024 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:51.694541 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:51.694489 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx"] Oct 31 01:12:51.741803 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:51.741758 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" Oct 31 01:12:51.749154 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:51.749120 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx"] Oct 31 01:12:51.813524 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:51.813493 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/961cf83c-7198-4f01-9a04-00d48bb61406-istio-token\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx\" (UID: \"961cf83c-7198-4f01-9a04-00d48bb61406\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" Oct 31 01:12:51.813693 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:51.813532 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/961cf83c-7198-4f01-9a04-00d48bb61406-istio-envoy\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx\" (UID: \"961cf83c-7198-4f01-9a04-00d48bb61406\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" Oct 31 01:12:51.813693 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:51.813616 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/961cf83c-7198-4f01-9a04-00d48bb61406-workload-socket\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx\" (UID: \"961cf83c-7198-4f01-9a04-00d48bb61406\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" Oct 31 01:12:51.813693 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:51.813667 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/961cf83c-7198-4f01-9a04-00d48bb61406-istiod-ca-cert\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx\" (UID: \"961cf83c-7198-4f01-9a04-00d48bb61406\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" Oct 31 01:12:51.813875 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:51.813723 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/961cf83c-7198-4f01-9a04-00d48bb61406-istio-podinfo\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx\" (UID: \"961cf83c-7198-4f01-9a04-00d48bb61406\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" Oct 31 01:12:51.813875 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:51.813862 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpp7v\" (UniqueName: \"kubernetes.io/projected/961cf83c-7198-4f01-9a04-00d48bb61406-kube-api-access-tpp7v\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx\" (UID: \"961cf83c-7198-4f01-9a04-00d48bb61406\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" Oct 31 01:12:51.813976 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:51.813887 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/961cf83c-7198-4f01-9a04-00d48bb61406-workload-certs\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx\" (UID: \"961cf83c-7198-4f01-9a04-00d48bb61406\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" Oct 31 01:12:51.814213 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:51.814185 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/961cf83c-7198-4f01-9a04-00d48bb61406-istio-data\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx\" (UID: \"961cf83c-7198-4f01-9a04-00d48bb61406\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" Oct 31 01:12:51.814356 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:51.814287 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/961cf83c-7198-4f01-9a04-00d48bb61406-credential-socket\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx\" (UID: \"961cf83c-7198-4f01-9a04-00d48bb61406\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" Oct 31 01:12:51.915410 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:51.915375 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/961cf83c-7198-4f01-9a04-00d48bb61406-istio-envoy\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx\" (UID: \"961cf83c-7198-4f01-9a04-00d48bb61406\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" Oct 31 01:12:51.915619 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:51.915460 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/961cf83c-7198-4f01-9a04-00d48bb61406-workload-socket\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx\" (UID: \"961cf83c-7198-4f01-9a04-00d48bb61406\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" Oct 31 01:12:51.915619 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:51.915504 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/961cf83c-7198-4f01-9a04-00d48bb61406-istiod-ca-cert\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx\" (UID: \"961cf83c-7198-4f01-9a04-00d48bb61406\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" Oct 31 01:12:51.915619 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:51.915549 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/961cf83c-7198-4f01-9a04-00d48bb61406-istio-podinfo\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx\" (UID: \"961cf83c-7198-4f01-9a04-00d48bb61406\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" Oct 31 01:12:51.915796 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:51.915714 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpp7v\" (UniqueName: \"kubernetes.io/projected/961cf83c-7198-4f01-9a04-00d48bb61406-kube-api-access-tpp7v\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx\" (UID: \"961cf83c-7198-4f01-9a04-00d48bb61406\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" Oct 31 01:12:51.915796 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:51.915741 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/961cf83c-7198-4f01-9a04-00d48bb61406-workload-certs\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx\" (UID: \"961cf83c-7198-4f01-9a04-00d48bb61406\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" Oct 31 01:12:51.915905 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:51.915814 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/961cf83c-7198-4f01-9a04-00d48bb61406-istio-data\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx\" (UID: \"961cf83c-7198-4f01-9a04-00d48bb61406\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" Oct 31 01:12:51.915905 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:51.915850 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/961cf83c-7198-4f01-9a04-00d48bb61406-credential-socket\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx\" (UID: \"961cf83c-7198-4f01-9a04-00d48bb61406\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" Oct 31 01:12:51.916014 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:51.915932 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/961cf83c-7198-4f01-9a04-00d48bb61406-workload-socket\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx\" (UID: \"961cf83c-7198-4f01-9a04-00d48bb61406\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" Oct 31 01:12:51.916354 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:51.916139 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/961cf83c-7198-4f01-9a04-00d48bb61406-istio-token\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx\" (UID: \"961cf83c-7198-4f01-9a04-00d48bb61406\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" Oct 31 01:12:51.916354 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:51.916244 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/961cf83c-7198-4f01-9a04-00d48bb61406-workload-certs\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx\" (UID: \"961cf83c-7198-4f01-9a04-00d48bb61406\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" Oct 31 01:12:51.916354 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:51.916276 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/961cf83c-7198-4f01-9a04-00d48bb61406-credential-socket\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx\" (UID: \"961cf83c-7198-4f01-9a04-00d48bb61406\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" Oct 31 01:12:51.916354 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:51.916317 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/961cf83c-7198-4f01-9a04-00d48bb61406-istio-data\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx\" (UID: \"961cf83c-7198-4f01-9a04-00d48bb61406\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" Oct 31 01:12:51.916354 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:51.916327 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/961cf83c-7198-4f01-9a04-00d48bb61406-istiod-ca-cert\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx\" (UID: \"961cf83c-7198-4f01-9a04-00d48bb61406\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" Oct 31 01:12:51.917983 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:51.917962 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/961cf83c-7198-4f01-9a04-00d48bb61406-istio-envoy\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx\" (UID: \"961cf83c-7198-4f01-9a04-00d48bb61406\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" Oct 31 01:12:51.918541 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:51.918525 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/961cf83c-7198-4f01-9a04-00d48bb61406-istio-podinfo\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx\" (UID: \"961cf83c-7198-4f01-9a04-00d48bb61406\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" Oct 31 01:12:51.923759 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:51.923728 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/961cf83c-7198-4f01-9a04-00d48bb61406-istio-token\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx\" (UID: \"961cf83c-7198-4f01-9a04-00d48bb61406\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" Oct 31 01:12:51.924030 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:51.924015 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpp7v\" (UniqueName: \"kubernetes.io/projected/961cf83c-7198-4f01-9a04-00d48bb61406-kube-api-access-tpp7v\") pod \"e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx\" (UID: \"961cf83c-7198-4f01-9a04-00d48bb61406\") " pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" Oct 31 01:12:52.051403 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:52.051307 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" Oct 31 01:12:52.066659 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:52.066623 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" Oct 31 01:12:52.370325 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:12:52.370290 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod961cf83c_7198_4f01_9a04_00d48bb61406.slice/crio-2842ff4b211268b472602e0a1ed7cf7bdc57fc41b58cc81c63c7ea9669faee39 WatchSource:0}: Error finding container 2842ff4b211268b472602e0a1ed7cf7bdc57fc41b58cc81c63c7ea9669faee39: Status 404 returned error can't find the container with id 2842ff4b211268b472602e0a1ed7cf7bdc57fc41b58cc81c63c7ea9669faee39 Oct 31 01:12:52.373231 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:52.373206 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx"] Oct 31 01:12:52.373311 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:52.373230 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:12:52.373353 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:52.373312 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:12:52.373386 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:52.373358 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:12:52.478457 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:52.476806 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:53.045021 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:53.044982 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" event={"ID":"961cf83c-7198-4f01-9a04-00d48bb61406","Type":"ContainerStarted","Data":"49ee265d5d7615a3f2f6d41e7ad9c12dbe984d1ff9a1b1cf834858150420f6b6"} Oct 31 01:12:53.045527 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:53.045050 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" event={"ID":"961cf83c-7198-4f01-9a04-00d48bb61406","Type":"ContainerStarted","Data":"2842ff4b211268b472602e0a1ed7cf7bdc57fc41b58cc81c63c7ea9669faee39"} Oct 31 01:12:53.067722 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:53.067689 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" Oct 31 01:12:53.069091 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:53.069044 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" podStartSLOduration=2.069031628 podStartE2EDuration="2.069031628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:53.065735024 +0000 UTC m=+1017.424457340" watchObservedRunningTime="2026-10-31 01:12:53.069031628 +0000 UTC m=+1017.427753944" Oct 31 01:12:53.069272 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:53.069252 4440 patch_prober.go:29] interesting pod/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 31 01:12:53.069315 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:53.069295 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" podUID="961cf83c-7198-4f01-9a04-00d48bb61406" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 31 01:12:53.195262 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:53.195230 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:54.067246 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:54.067213 4440 patch_prober.go:29] interesting pod/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 31 01:12:54.067715 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:54.067265 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" podUID="961cf83c-7198-4f01-9a04-00d48bb61406" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 31 01:12:54.244427 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:54.244387 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:55.067771 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:55.067737 4440 patch_prober.go:29] interesting pod/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 31 01:12:55.068241 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:55.067807 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" podUID="961cf83c-7198-4f01-9a04-00d48bb61406" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 31 01:12:55.316443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:55.314899 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:56.067202 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:56.067168 4440 patch_prober.go:29] interesting pod/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 31 01:12:56.067382 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:56.067225 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" podUID="961cf83c-7198-4f01-9a04-00d48bb61406" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 31 01:12:56.323638 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:56.323553 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:57.067058 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:57.067019 4440 patch_prober.go:29] interesting pod/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 31 01:12:57.067224 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:57.067094 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" podUID="961cf83c-7198-4f01-9a04-00d48bb61406" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 31 01:12:57.450379 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:57.450343 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:57.550605 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:57.550503 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx"] Oct 31 01:12:57.557019 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:57.556993 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx"] Oct 31 01:12:57.557194 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:57.557083 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" Oct 31 01:12:57.561636 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:57.561612 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8499c8\"/\"distinct-2-w4-8499c8-kserve-self-signed-certs\"" Oct 31 01:12:57.586132 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:57.586105 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:57.740158 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:57.740011 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfz49\" (UniqueName: \"kubernetes.io/projected/2f72d419-0877-4e49-9a85-095ddea8c790-kube-api-access-rfz49\") pod \"distinct-2-w4-8499c8-kserve-778bf48555-wpxwx\" (UID: \"2f72d419-0877-4e49-9a85-095ddea8c790\") " pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" Oct 31 01:12:57.740518 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:57.740490 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f72d419-0877-4e49-9a85-095ddea8c790-tmp-dir\") pod \"distinct-2-w4-8499c8-kserve-778bf48555-wpxwx\" (UID: \"2f72d419-0877-4e49-9a85-095ddea8c790\") " pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" Oct 31 01:12:57.740750 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:57.740734 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f72d419-0877-4e49-9a85-095ddea8c790-home\") pod \"distinct-2-w4-8499c8-kserve-778bf48555-wpxwx\" (UID: \"2f72d419-0877-4e49-9a85-095ddea8c790\") " pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" Oct 31 01:12:57.740950 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:57.740931 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f72d419-0877-4e49-9a85-095ddea8c790-tls-certs\") pod \"distinct-2-w4-8499c8-kserve-778bf48555-wpxwx\" (UID: \"2f72d419-0877-4e49-9a85-095ddea8c790\") " pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" Oct 31 01:12:57.741316 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:57.741294 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f72d419-0877-4e49-9a85-095ddea8c790-model-cache\") pod \"distinct-2-w4-8499c8-kserve-778bf48555-wpxwx\" (UID: \"2f72d419-0877-4e49-9a85-095ddea8c790\") " pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" Oct 31 01:12:57.741528 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:57.741509 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f72d419-0877-4e49-9a85-095ddea8c790-dshm\") pod \"distinct-2-w4-8499c8-kserve-778bf48555-wpxwx\" (UID: \"2f72d419-0877-4e49-9a85-095ddea8c790\") " pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" Oct 31 01:12:57.842541 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:57.842510 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f72d419-0877-4e49-9a85-095ddea8c790-model-cache\") pod \"distinct-2-w4-8499c8-kserve-778bf48555-wpxwx\" (UID: \"2f72d419-0877-4e49-9a85-095ddea8c790\") " pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" Oct 31 01:12:57.843145 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:57.843124 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f72d419-0877-4e49-9a85-095ddea8c790-dshm\") pod \"distinct-2-w4-8499c8-kserve-778bf48555-wpxwx\" (UID: \"2f72d419-0877-4e49-9a85-095ddea8c790\") " pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" Oct 31 01:12:57.843315 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:57.843302 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfz49\" (UniqueName: \"kubernetes.io/projected/2f72d419-0877-4e49-9a85-095ddea8c790-kube-api-access-rfz49\") pod \"distinct-2-w4-8499c8-kserve-778bf48555-wpxwx\" (UID: \"2f72d419-0877-4e49-9a85-095ddea8c790\") " pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" Oct 31 01:12:57.843403 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:57.843390 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f72d419-0877-4e49-9a85-095ddea8c790-tmp-dir\") pod \"distinct-2-w4-8499c8-kserve-778bf48555-wpxwx\" (UID: \"2f72d419-0877-4e49-9a85-095ddea8c790\") " pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" Oct 31 01:12:57.843546 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:57.843535 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f72d419-0877-4e49-9a85-095ddea8c790-home\") pod \"distinct-2-w4-8499c8-kserve-778bf48555-wpxwx\" (UID: \"2f72d419-0877-4e49-9a85-095ddea8c790\") " pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" Oct 31 01:12:57.843620 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:57.843610 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f72d419-0877-4e49-9a85-095ddea8c790-tls-certs\") pod \"distinct-2-w4-8499c8-kserve-778bf48555-wpxwx\" (UID: \"2f72d419-0877-4e49-9a85-095ddea8c790\") " pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" Oct 31 01:12:57.844762 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:57.843054 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f72d419-0877-4e49-9a85-095ddea8c790-model-cache\") pod \"distinct-2-w4-8499c8-kserve-778bf48555-wpxwx\" (UID: \"2f72d419-0877-4e49-9a85-095ddea8c790\") " pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" Oct 31 01:12:57.852185 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:57.852162 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f72d419-0877-4e49-9a85-095ddea8c790-tmp-dir\") pod \"distinct-2-w4-8499c8-kserve-778bf48555-wpxwx\" (UID: \"2f72d419-0877-4e49-9a85-095ddea8c790\") " pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" Oct 31 01:12:57.852579 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:57.852561 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f72d419-0877-4e49-9a85-095ddea8c790-home\") pod \"distinct-2-w4-8499c8-kserve-778bf48555-wpxwx\" (UID: \"2f72d419-0877-4e49-9a85-095ddea8c790\") " pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" Oct 31 01:12:57.859966 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:57.859368 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f72d419-0877-4e49-9a85-095ddea8c790-tls-certs\") pod \"distinct-2-w4-8499c8-kserve-778bf48555-wpxwx\" (UID: \"2f72d419-0877-4e49-9a85-095ddea8c790\") " pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" Oct 31 01:12:57.859966 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:57.859646 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f72d419-0877-4e49-9a85-095ddea8c790-dshm\") pod \"distinct-2-w4-8499c8-kserve-778bf48555-wpxwx\" (UID: \"2f72d419-0877-4e49-9a85-095ddea8c790\") " pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" Oct 31 01:12:57.860213 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:57.860128 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfz49\" (UniqueName: \"kubernetes.io/projected/2f72d419-0877-4e49-9a85-095ddea8c790-kube-api-access-rfz49\") pod \"distinct-2-w4-8499c8-kserve-778bf48555-wpxwx\" (UID: \"2f72d419-0877-4e49-9a85-095ddea8c790\") " pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" Oct 31 01:12:57.874705 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:57.874677 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" Oct 31 01:12:58.067315 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:58.067154 4440 patch_prober.go:29] interesting pod/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 31 01:12:58.067315 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:58.067205 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" podUID="961cf83c-7198-4f01-9a04-00d48bb61406" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 31 01:12:58.256461 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:58.255361 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx"] Oct 31 01:12:58.258973 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:12:58.258938 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f72d419_0877_4e49_9a85_095ddea8c790.slice/crio-76eb3fb62f49e71ec6002a9edfa1219d0ea07306c08c9f9742ef5a16f72bd9f5 WatchSource:0}: Error finding container 76eb3fb62f49e71ec6002a9edfa1219d0ea07306c08c9f9742ef5a16f72bd9f5: Status 404 returned error can't find the container with id 76eb3fb62f49e71ec6002a9edfa1219d0ea07306c08c9f9742ef5a16f72bd9f5 Oct 31 01:12:58.822437 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:58.819917 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:59.067033 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:59.066994 4440 patch_prober.go:29] interesting pod/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 31 01:12:59.067188 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:59.067059 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" podUID="961cf83c-7198-4f01-9a04-00d48bb61406" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 31 01:12:59.091508 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:59.091412 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" event={"ID":"2f72d419-0877-4e49-9a85-095ddea8c790","Type":"ContainerStarted","Data":"2379083646d1c3ced10003652094f928ee54c73768e44bb54a990a7dd059da98"} Oct 31 01:12:59.091508 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:59.091468 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" event={"ID":"2f72d419-0877-4e49-9a85-095ddea8c790","Type":"ContainerStarted","Data":"76eb3fb62f49e71ec6002a9edfa1219d0ea07306c08c9f9742ef5a16f72bd9f5"} Oct 31 01:12:59.109337 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:59.109274 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" podStartSLOduration=2.10925531 podStartE2EDuration="2.10925531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:12:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:59.10725255 +0000 UTC m=+1023.465974865" watchObservedRunningTime="2026-10-31 01:12:59.10925531 +0000 UTC m=+1023.467977626" Oct 31 01:12:59.572499 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:59.572455 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:59.875053 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:12:59.874964 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" Oct 31 01:13:00.072334 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:00.072297 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" Oct 31 01:13:00.073116 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:00.073084 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" Oct 31 01:13:00.074017 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:00.073995 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" Oct 31 01:13:00.375565 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:00.375522 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:01.019925 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:13:01.019812 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:13:01.215918 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:01.215826 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:01.850643 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:01.850610 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:02.673907 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:02.673865 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:03.385158 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:03.385119 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:05.690445 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:05.690296 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:05.866134 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:05.864998 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 01:13:05.940614 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:05.940502 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:06.081756 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:06.081704 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" podUID="cf86081d-7942-454a-8763-f658bd0ce870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:06.243840 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:06.242895 4440 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:06.354638 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:06.354570 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 31 01:13:06.397975 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:06.397920 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:06.693441 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:06.691217 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:06.693441 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:06.691976 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" Oct 31 01:13:06.701573 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:06.694959 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:06.868802 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:06.868749 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:06.917737 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:06.903804 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" Oct 31 01:13:06.931442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:06.927850 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" podUID="617e420d-d029-4f4b-8ae5-63fe7d85971e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:07.036613 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:07.036481 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:07.226601 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:07.226556 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" podUID="0b82b50e-bfa7-45b9-8484-c7a331b5ac03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:07.227344 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:07.226873 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:07.244442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:07.226924 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" podUID="cf86081d-7942-454a-8763-f658bd0ce870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:07.310619 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:07.248549 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:07.355924 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:07.355888 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" Oct 31 01:13:07.363442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:07.356243 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" Oct 31 01:13:07.506441 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:07.486738 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:07.525593 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:07.488382 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 31 01:13:07.550966 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:07.550929 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" Oct 31 01:13:07.599563 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:07.594538 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" Oct 31 01:13:07.642565 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:07.620985 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz"] Oct 31 01:13:07.719881 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:07.719832 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:08.053441 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:07.761494 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 31 01:13:08.053441 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:07.922542 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:08.053441 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:07.922679 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" Oct 31 01:13:08.074446 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:08.009333 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" podUID="617e420d-d029-4f4b-8ae5-63fe7d85971e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": context deadline exceeded" Oct 31 01:13:08.075232 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:08.037399 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:08.239783 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:08.239736 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 31 01:13:08.240847 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:08.240817 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" podUID="0b82b50e-bfa7-45b9-8484-c7a331b5ac03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 31 01:13:08.516642 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:08.278539 4440 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" podUID="0b82b50e-bfa7-45b9-8484-c7a331b5ac03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": context deadline exceeded" Oct 31 01:13:08.516968 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:08.362496 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" podUID="cf86081d-7942-454a-8763-f658bd0ce870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:08.886474 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:08.517395 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:08.902494 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:08.548663 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ef7da0-2a19-4794-a80f-9badb9461e9a-workload-socket\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz\" (UID: \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" Oct 31 01:13:08.902494 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:08.889238 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmk9b\" (UniqueName: \"kubernetes.io/projected/a2ef7da0-2a19-4794-a80f-9badb9461e9a-kube-api-access-zmk9b\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz\" (UID: \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" Oct 31 01:13:08.902494 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:08.889323 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ef7da0-2a19-4794-a80f-9badb9461e9a-credential-socket\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz\" (UID: \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" Oct 31 01:13:08.902494 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:08.889386 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2ef7da0-2a19-4794-a80f-9badb9461e9a-workload-certs\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz\" (UID: \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" Oct 31 01:13:08.902494 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:08.889669 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2ef7da0-2a19-4794-a80f-9badb9461e9a-istio-token\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz\" (UID: \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" Oct 31 01:13:08.902494 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:08.889709 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2ef7da0-2a19-4794-a80f-9badb9461e9a-istio-envoy\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz\" (UID: \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" Oct 31 01:13:08.902494 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:08.889854 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2ef7da0-2a19-4794-a80f-9badb9461e9a-istio-data\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz\" (UID: \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" Oct 31 01:13:08.902494 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:08.890364 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2ef7da0-2a19-4794-a80f-9badb9461e9a-istio-podinfo\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz\" (UID: \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" Oct 31 01:13:08.906685 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:08.906608 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2ef7da0-2a19-4794-a80f-9badb9461e9a-istiod-ca-cert\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz\" (UID: \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" Oct 31 01:13:08.990443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:08.550064 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 31 01:13:08.990443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:08.550757 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:08.990443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:08.579532 4440 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:09.097387 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:09.097339 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ef7da0-2a19-4794-a80f-9badb9461e9a-credential-socket\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz\" (UID: \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" Oct 31 01:13:09.104647 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:09.104593 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2ef7da0-2a19-4794-a80f-9badb9461e9a-workload-certs\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz\" (UID: \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" Oct 31 01:13:09.119454 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:09.119351 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2ef7da0-2a19-4794-a80f-9badb9461e9a-istio-token\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz\" (UID: \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" Oct 31 01:13:09.135487 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:09.134518 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2ef7da0-2a19-4794-a80f-9badb9461e9a-istio-envoy\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz\" (UID: \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" Oct 31 01:13:09.135487 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:09.134721 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2ef7da0-2a19-4794-a80f-9badb9461e9a-istio-data\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz\" (UID: \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" Oct 31 01:13:09.135487 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:09.134923 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2ef7da0-2a19-4794-a80f-9badb9461e9a-istio-podinfo\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz\" (UID: \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" Oct 31 01:13:09.135487 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:09.135066 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2ef7da0-2a19-4794-a80f-9badb9461e9a-istiod-ca-cert\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz\" (UID: \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" Oct 31 01:13:09.135487 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:09.135247 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ef7da0-2a19-4794-a80f-9badb9461e9a-workload-socket\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz\" (UID: \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" Oct 31 01:13:09.135487 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:09.135318 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmk9b\" (UniqueName: \"kubernetes.io/projected/a2ef7da0-2a19-4794-a80f-9badb9461e9a-kube-api-access-zmk9b\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz\" (UID: \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" Oct 31 01:13:09.136816 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:09.136750 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ef7da0-2a19-4794-a80f-9badb9461e9a-credential-socket\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz\" (UID: \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" Oct 31 01:13:09.156346 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:09.155828 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2ef7da0-2a19-4794-a80f-9badb9461e9a-workload-certs\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz\" (UID: \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" Oct 31 01:13:09.319828 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:09.319756 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:09.342219 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:13:09.337925 4440 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.471s" Oct 31 01:13:09.439446 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:09.389137 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" Oct 31 01:13:09.482480 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:09.468830 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz"] Oct 31 01:13:09.482480 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:09.469377 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" Oct 31 01:13:09.482480 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:09.469508 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" Oct 31 01:13:09.482480 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:09.469892 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" Oct 31 01:13:09.482480 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:09.469986 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" Oct 31 01:13:09.504582 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:09.417227 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ef7da0-2a19-4794-a80f-9badb9461e9a-workload-socket\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz\" (UID: \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" Oct 31 01:13:09.519396 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:09.519354 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2ef7da0-2a19-4794-a80f-9badb9461e9a-istio-token\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz\" (UID: \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" Oct 31 01:13:09.543747 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:09.543698 4440 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:09.582108 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:09.461133 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2ef7da0-2a19-4794-a80f-9badb9461e9a-istio-data\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz\" (UID: \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" Oct 31 01:13:09.593447 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:09.465376 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2ef7da0-2a19-4794-a80f-9badb9461e9a-istiod-ca-cert\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz\" (UID: \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" Oct 31 01:13:09.645499 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:09.591331 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2ef7da0-2a19-4794-a80f-9badb9461e9a-istio-envoy\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz\" (UID: \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" Oct 31 01:13:09.645757 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:09.604480 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" podUID="0b82b50e-bfa7-45b9-8484-c7a331b5ac03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:09.696498 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:09.683504 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2ef7da0-2a19-4794-a80f-9badb9461e9a-istio-podinfo\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz\" (UID: \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" Oct 31 01:13:09.821587 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:09.821531 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" podUID="617e420d-d029-4f4b-8ae5-63fe7d85971e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": context deadline exceeded" Oct 31 01:13:09.950813 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:09.897976 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" podUID="cf86081d-7942-454a-8763-f658bd0ce870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 31 01:13:09.960066 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:09.908785 4440 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" podUID="2f72d419-0877-4e49-9a85-095ddea8c790" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": context deadline exceeded" Oct 31 01:13:09.960342 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:09.909152 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 31 01:13:09.985481 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:09.949156 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:09.986055 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:09.949192 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 31 01:13:09.986310 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:09.949225 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 31 01:13:10.171456 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:10.149908 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 31 01:13:10.351510 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:10.348326 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 01:13:10.360518 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:10.191930 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 01:13:10.566097 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:10.393181 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmk9b\" (UniqueName: \"kubernetes.io/projected/a2ef7da0-2a19-4794-a80f-9badb9461e9a-kube-api-access-zmk9b\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz\" (UID: \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" Oct 31 01:13:10.566097 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:10.433017 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 31 01:13:10.702450 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:10.691489 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kr4hj" podUID="76e1f133-7f40-4472-a3f3-c1b6944818d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:10.702450 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:10.694563 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" podUID="0b82b50e-bfa7-45b9-8484-c7a331b5ac03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 31 01:13:10.856444 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:10.850846 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tvltr" podUID="c14a3342-f1ee-4856-97ff-4c0cc1e25e19" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:10.856444 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:10.855659 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" podUID="617e420d-d029-4f4b-8ae5-63fe7d85971e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": context deadline exceeded" Oct 31 01:13:10.870444 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:10.866578 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" Oct 31 01:13:11.065499 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:11.065380 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:11.066354 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:11.066324 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" podUID="cf86081d-7942-454a-8763-f658bd0ce870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 31 01:13:11.066975 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:11.066950 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 31 01:13:11.067510 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:11.067488 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 31 01:13:11.069903 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:11.069865 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 31 01:13:11.160092 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:11.154242 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:11.309440 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:11.298690 4440 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:11.365503 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:11.364603 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:11.365503 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:11.365294 4440 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:11.496444 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:11.491768 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:11.611514 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:11.611466 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:11.732431 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:11.731027 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" podUID="0b82b50e-bfa7-45b9-8484-c7a331b5ac03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:11.903499 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:11.871494 4440 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:11.920960 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:11.920919 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" podUID="617e420d-d029-4f4b-8ae5-63fe7d85971e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": context deadline exceeded" Oct 31 01:13:11.935364 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:13:11.935333 4440 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.473s" Oct 31 01:13:11.995593 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:11.995451 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" Oct 31 01:13:11.998358 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:11.998335 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" Oct 31 01:13:11.999345 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:11.998973 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" Oct 31 01:13:12.151529 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:12.150547 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 31 01:13:12.151529 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:12.150894 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" podUID="cf86081d-7942-454a-8763-f658bd0ce870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 31 01:13:12.152050 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:12.151645 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 31 01:13:12.152050 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:12.151909 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 31 01:13:12.153274 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:12.153078 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:12.162435 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:12.154688 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 31 01:13:12.165577 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:12.165557 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" Oct 31 01:13:12.183044 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:12.165851 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:12.189724 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:12.189695 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" Oct 31 01:13:12.191263 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:12.191230 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" Oct 31 01:13:12.193050 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:12.192854 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" Oct 31 01:13:12.205813 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:12.205782 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" Oct 31 01:13:12.221463 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:12.217104 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" Oct 31 01:13:12.221463 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:12.217314 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" Oct 31 01:13:12.223890 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:12.223864 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" Oct 31 01:13:12.225183 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:12.225116 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" Oct 31 01:13:12.474850 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:12.474823 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz"] Oct 31 01:13:12.479882 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:13:12.479843 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2ef7da0_2a19_4794_a80f_9badb9461e9a.slice/crio-40b3e1ae3ae7b637e60abd854227198bd015650732fdfcf4cdb1f25508f06a7c WatchSource:0}: Error finding container 40b3e1ae3ae7b637e60abd854227198bd015650732fdfcf4cdb1f25508f06a7c: Status 404 returned error can't find the container with id 40b3e1ae3ae7b637e60abd854227198bd015650732fdfcf4cdb1f25508f06a7c Oct 31 01:13:12.493006 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:12.492967 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:13:12.493149 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:12.493045 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:13:12.493149 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:12.493082 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:13:13.208511 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:13.208060 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:13.455748 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:13.455709 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" event={"ID":"a2ef7da0-2a19-4794-a80f-9badb9461e9a","Type":"ContainerStarted","Data":"29979f9137e8c116558b7b16cfac14685809a3092cad8481902db8d834367e07"} Oct 31 01:13:13.455929 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:13.455770 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" event={"ID":"a2ef7da0-2a19-4794-a80f-9badb9461e9a","Type":"ContainerStarted","Data":"40b3e1ae3ae7b637e60abd854227198bd015650732fdfcf4cdb1f25508f06a7c"} Oct 31 01:13:13.482754 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:13.482646 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" podStartSLOduration=6.482631798 podStartE2EDuration="6.482631798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:13.478095117 +0000 UTC m=+1037.836817436" watchObservedRunningTime="2026-10-31 01:13:13.482631798 +0000 UTC m=+1037.841354115" Oct 31 01:13:13.893222 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:13.893125 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" Oct 31 01:13:13.894801 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:13.894771 4440 patch_prober.go:29] interesting pod/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 31 01:13:13.894936 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:13.894823 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" podUID="a2ef7da0-2a19-4794-a80f-9badb9461e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 31 01:13:13.919440 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:13.918659 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:14.673386 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:14.673347 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:14.893491 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:14.893409 4440 patch_prober.go:29] interesting pod/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 31 01:13:14.893680 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:14.893520 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" podUID="a2ef7da0-2a19-4794-a80f-9badb9461e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 31 01:13:15.405152 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:15.405111 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:15.893832 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:15.893797 4440 patch_prober.go:29] interesting pod/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 31 01:13:15.894287 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:15.893853 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" podUID="a2ef7da0-2a19-4794-a80f-9badb9461e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 31 01:13:16.181241 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:16.181205 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:16.894013 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:16.893957 4440 patch_prober.go:29] interesting pod/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 31 01:13:16.894520 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:16.894048 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" podUID="a2ef7da0-2a19-4794-a80f-9badb9461e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 31 01:13:17.010409 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:17.010370 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:17.732968 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:17.732927 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:17.895118 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:17.894702 4440 patch_prober.go:29] interesting pod/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 31 01:13:17.895118 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:17.894772 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" podUID="a2ef7da0-2a19-4794-a80f-9badb9461e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 31 01:13:17.899786 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:17.895835 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" Oct 31 01:13:17.928026 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:17.926727 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" Oct 31 01:13:18.481387 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:18.481352 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:18.894084 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:18.893997 4440 patch_prober.go:29] interesting pod/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 31 01:13:18.894084 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:18.894054 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" podUID="a2ef7da0-2a19-4794-a80f-9badb9461e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 31 01:13:19.510211 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:19.510160 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:20.152794 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:20.152753 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" Oct 31 01:13:20.153669 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:20.153641 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" Oct 31 01:13:20.154557 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:20.154534 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" Oct 31 01:13:20.382466 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:20.382379 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:21.282783 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:21.282745 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:22.034748 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.034712 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br"] Oct 31 01:13:22.040296 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.040265 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" Oct 31 01:13:22.054254 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.054225 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br"] Oct 31 01:13:22.142507 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.142461 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b984db-1113-44af-b6d9-6cf452c97187-credential-socket\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br\" (UID: \"66b984db-1113-44af-b6d9-6cf452c97187\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" Oct 31 01:13:22.142760 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.142550 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66b984db-1113-44af-b6d9-6cf452c97187-istio-token\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br\" (UID: \"66b984db-1113-44af-b6d9-6cf452c97187\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" Oct 31 01:13:22.142760 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.142612 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b984db-1113-44af-b6d9-6cf452c97187-workload-socket\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br\" (UID: \"66b984db-1113-44af-b6d9-6cf452c97187\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" Oct 31 01:13:22.142760 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.142690 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psdp9\" (UniqueName: \"kubernetes.io/projected/66b984db-1113-44af-b6d9-6cf452c97187-kube-api-access-psdp9\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br\" (UID: \"66b984db-1113-44af-b6d9-6cf452c97187\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" Oct 31 01:13:22.142760 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.142732 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66b984db-1113-44af-b6d9-6cf452c97187-workload-certs\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br\" (UID: \"66b984db-1113-44af-b6d9-6cf452c97187\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" Oct 31 01:13:22.143086 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.142779 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66b984db-1113-44af-b6d9-6cf452c97187-istio-envoy\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br\" (UID: \"66b984db-1113-44af-b6d9-6cf452c97187\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" Oct 31 01:13:22.143086 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.142811 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66b984db-1113-44af-b6d9-6cf452c97187-istio-data\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br\" (UID: \"66b984db-1113-44af-b6d9-6cf452c97187\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" Oct 31 01:13:22.143086 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.142850 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66b984db-1113-44af-b6d9-6cf452c97187-istiod-ca-cert\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br\" (UID: \"66b984db-1113-44af-b6d9-6cf452c97187\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" Oct 31 01:13:22.143086 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.142928 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66b984db-1113-44af-b6d9-6cf452c97187-istio-podinfo\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br\" (UID: \"66b984db-1113-44af-b6d9-6cf452c97187\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" Oct 31 01:13:22.153352 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.153318 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:22.217026 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.216986 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5"] Oct 31 01:13:22.222702 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.222674 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" Oct 31 01:13:22.232893 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.232862 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5"] Oct 31 01:13:22.244319 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.244281 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b984db-1113-44af-b6d9-6cf452c97187-credential-socket\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br\" (UID: \"66b984db-1113-44af-b6d9-6cf452c97187\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" Oct 31 01:13:22.244509 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.244340 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f889944d-e872-4b84-baa6-829de12f3ba1-istio-data\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5\" (UID: \"f889944d-e872-4b84-baa6-829de12f3ba1\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" Oct 31 01:13:22.244509 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.244368 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f889944d-e872-4b84-baa6-829de12f3ba1-istio-token\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5\" (UID: \"f889944d-e872-4b84-baa6-829de12f3ba1\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" Oct 31 01:13:22.244651 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.244533 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66b984db-1113-44af-b6d9-6cf452c97187-istio-token\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br\" (UID: \"66b984db-1113-44af-b6d9-6cf452c97187\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" Oct 31 01:13:22.244651 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.244607 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b984db-1113-44af-b6d9-6cf452c97187-workload-socket\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br\" (UID: \"66b984db-1113-44af-b6d9-6cf452c97187\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" Oct 31 01:13:22.244766 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.244651 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f889944d-e872-4b84-baa6-829de12f3ba1-istiod-ca-cert\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5\" (UID: \"f889944d-e872-4b84-baa6-829de12f3ba1\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" Oct 31 01:13:22.244766 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.244703 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f889944d-e872-4b84-baa6-829de12f3ba1-credential-socket\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5\" (UID: \"f889944d-e872-4b84-baa6-829de12f3ba1\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" Oct 31 01:13:22.244766 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.244756 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psdp9\" (UniqueName: \"kubernetes.io/projected/66b984db-1113-44af-b6d9-6cf452c97187-kube-api-access-psdp9\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br\" (UID: \"66b984db-1113-44af-b6d9-6cf452c97187\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" Oct 31 01:13:22.244924 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.244796 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f889944d-e872-4b84-baa6-829de12f3ba1-istio-podinfo\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5\" (UID: \"f889944d-e872-4b84-baa6-829de12f3ba1\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" Oct 31 01:13:22.244924 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.244828 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66b984db-1113-44af-b6d9-6cf452c97187-workload-certs\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br\" (UID: \"66b984db-1113-44af-b6d9-6cf452c97187\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" Oct 31 01:13:22.244924 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.244885 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66b984db-1113-44af-b6d9-6cf452c97187-istio-envoy\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br\" (UID: \"66b984db-1113-44af-b6d9-6cf452c97187\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" Oct 31 01:13:22.245078 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.244922 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66b984db-1113-44af-b6d9-6cf452c97187-istio-data\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br\" (UID: \"66b984db-1113-44af-b6d9-6cf452c97187\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" Oct 31 01:13:22.245078 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.244971 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66b984db-1113-44af-b6d9-6cf452c97187-istiod-ca-cert\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br\" (UID: \"66b984db-1113-44af-b6d9-6cf452c97187\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" Oct 31 01:13:22.245078 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.245066 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66b984db-1113-44af-b6d9-6cf452c97187-istio-podinfo\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br\" (UID: \"66b984db-1113-44af-b6d9-6cf452c97187\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" Oct 31 01:13:22.245287 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.245152 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f889944d-e872-4b84-baa6-829de12f3ba1-workload-certs\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5\" (UID: \"f889944d-e872-4b84-baa6-829de12f3ba1\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" Oct 31 01:13:22.245287 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.245176 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f889944d-e872-4b84-baa6-829de12f3ba1-istio-envoy\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5\" (UID: \"f889944d-e872-4b84-baa6-829de12f3ba1\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" Oct 31 01:13:22.245287 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.245228 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9586g\" (UniqueName: \"kubernetes.io/projected/f889944d-e872-4b84-baa6-829de12f3ba1-kube-api-access-9586g\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5\" (UID: \"f889944d-e872-4b84-baa6-829de12f3ba1\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" Oct 31 01:13:22.245287 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.245280 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f889944d-e872-4b84-baa6-829de12f3ba1-workload-socket\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5\" (UID: \"f889944d-e872-4b84-baa6-829de12f3ba1\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" Oct 31 01:13:22.246508 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.246478 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b984db-1113-44af-b6d9-6cf452c97187-workload-socket\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br\" (UID: \"66b984db-1113-44af-b6d9-6cf452c97187\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" Oct 31 01:13:22.247921 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.247881 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66b984db-1113-44af-b6d9-6cf452c97187-istiod-ca-cert\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br\" (UID: \"66b984db-1113-44af-b6d9-6cf452c97187\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" Oct 31 01:13:22.248174 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.248147 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66b984db-1113-44af-b6d9-6cf452c97187-workload-certs\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br\" (UID: \"66b984db-1113-44af-b6d9-6cf452c97187\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" Oct 31 01:13:22.248840 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.248812 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b984db-1113-44af-b6d9-6cf452c97187-credential-socket\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br\" (UID: \"66b984db-1113-44af-b6d9-6cf452c97187\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" Oct 31 01:13:22.249297 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.249269 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66b984db-1113-44af-b6d9-6cf452c97187-istio-data\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br\" (UID: \"66b984db-1113-44af-b6d9-6cf452c97187\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" Oct 31 01:13:22.261268 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.261238 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66b984db-1113-44af-b6d9-6cf452c97187-istio-envoy\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br\" (UID: \"66b984db-1113-44af-b6d9-6cf452c97187\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" Oct 31 01:13:22.264034 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.264006 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66b984db-1113-44af-b6d9-6cf452c97187-istio-podinfo\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br\" (UID: \"66b984db-1113-44af-b6d9-6cf452c97187\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" Oct 31 01:13:22.266882 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.266850 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psdp9\" (UniqueName: \"kubernetes.io/projected/66b984db-1113-44af-b6d9-6cf452c97187-kube-api-access-psdp9\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br\" (UID: \"66b984db-1113-44af-b6d9-6cf452c97187\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" Oct 31 01:13:22.276282 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.275181 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq"] Oct 31 01:13:22.293522 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.289547 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66b984db-1113-44af-b6d9-6cf452c97187-istio-token\") pod \"e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br\" (UID: \"66b984db-1113-44af-b6d9-6cf452c97187\") " pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" Oct 31 01:13:22.293522 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.293148 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9"] Oct 31 01:13:22.297899 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.295287 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" Oct 31 01:13:22.304469 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.301940 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq"] Oct 31 01:13:22.304469 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.302029 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" Oct 31 01:13:22.304469 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.302170 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9"] Oct 31 01:13:22.347006 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.346971 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/117032f9-b742-432f-ac5f-70c9982b919f-credential-socket\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9\" (UID: \"117032f9-b742-432f-ac5f-70c9982b919f\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" Oct 31 01:13:22.347173 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.347028 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-istio-envoy\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq\" (UID: \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" Oct 31 01:13:22.347173 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.347071 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f889944d-e872-4b84-baa6-829de12f3ba1-istiod-ca-cert\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5\" (UID: \"f889944d-e872-4b84-baa6-829de12f3ba1\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" Oct 31 01:13:22.347905 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.347881 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f889944d-e872-4b84-baa6-829de12f3ba1-istiod-ca-cert\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5\" (UID: \"f889944d-e872-4b84-baa6-829de12f3ba1\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" Oct 31 01:13:22.348001 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.347947 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/117032f9-b742-432f-ac5f-70c9982b919f-istio-token\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9\" (UID: \"117032f9-b742-432f-ac5f-70c9982b919f\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" Oct 31 01:13:22.348061 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.348050 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f889944d-e872-4b84-baa6-829de12f3ba1-credential-socket\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5\" (UID: \"f889944d-e872-4b84-baa6-829de12f3ba1\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" Oct 31 01:13:22.348120 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.348083 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-istiod-ca-cert\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq\" (UID: \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" Oct 31 01:13:22.348175 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.348139 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/117032f9-b742-432f-ac5f-70c9982b919f-istiod-ca-cert\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9\" (UID: \"117032f9-b742-432f-ac5f-70c9982b919f\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" Oct 31 01:13:22.348175 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.348165 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-workload-socket\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq\" (UID: \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" Oct 31 01:13:22.348277 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.348194 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f889944d-e872-4b84-baa6-829de12f3ba1-istio-podinfo\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5\" (UID: \"f889944d-e872-4b84-baa6-829de12f3ba1\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" Oct 31 01:13:22.348277 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.348240 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-workload-certs\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq\" (UID: \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" Oct 31 01:13:22.348382 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.348283 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/117032f9-b742-432f-ac5f-70c9982b919f-istio-envoy\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9\" (UID: \"117032f9-b742-432f-ac5f-70c9982b919f\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" Oct 31 01:13:22.348581 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.348558 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/117032f9-b742-432f-ac5f-70c9982b919f-istio-data\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9\" (UID: \"117032f9-b742-432f-ac5f-70c9982b919f\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" Oct 31 01:13:22.348724 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.348695 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-credential-socket\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq\" (UID: \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" Oct 31 01:13:22.348781 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.348722 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f889944d-e872-4b84-baa6-829de12f3ba1-credential-socket\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5\" (UID: \"f889944d-e872-4b84-baa6-829de12f3ba1\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" Oct 31 01:13:22.348844 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.348808 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-istio-podinfo\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq\" (UID: \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" Oct 31 01:13:22.348895 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.348861 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/117032f9-b742-432f-ac5f-70c9982b919f-workload-socket\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9\" (UID: \"117032f9-b742-432f-ac5f-70c9982b919f\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" Oct 31 01:13:22.348948 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.348890 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8j59\" (UniqueName: \"kubernetes.io/projected/117032f9-b742-432f-ac5f-70c9982b919f-kube-api-access-v8j59\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9\" (UID: \"117032f9-b742-432f-ac5f-70c9982b919f\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" Oct 31 01:13:22.349000 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.348983 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk6vb\" (UniqueName: \"kubernetes.io/projected/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-kube-api-access-bk6vb\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq\" (UID: \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" Oct 31 01:13:22.349052 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.349034 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f889944d-e872-4b84-baa6-829de12f3ba1-workload-certs\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5\" (UID: \"f889944d-e872-4b84-baa6-829de12f3ba1\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" Oct 31 01:13:22.349108 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.349059 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f889944d-e872-4b84-baa6-829de12f3ba1-istio-envoy\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5\" (UID: \"f889944d-e872-4b84-baa6-829de12f3ba1\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" Oct 31 01:13:22.349161 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.349139 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9586g\" (UniqueName: \"kubernetes.io/projected/f889944d-e872-4b84-baa6-829de12f3ba1-kube-api-access-9586g\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5\" (UID: \"f889944d-e872-4b84-baa6-829de12f3ba1\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" Oct 31 01:13:22.349214 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.349183 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/117032f9-b742-432f-ac5f-70c9982b919f-istio-podinfo\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9\" (UID: \"117032f9-b742-432f-ac5f-70c9982b919f\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" Oct 31 01:13:22.349268 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.349209 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-istio-token\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq\" (UID: \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" Oct 31 01:13:22.349268 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.349245 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f889944d-e872-4b84-baa6-829de12f3ba1-workload-socket\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5\" (UID: \"f889944d-e872-4b84-baa6-829de12f3ba1\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" Oct 31 01:13:22.349498 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.349292 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/117032f9-b742-432f-ac5f-70c9982b919f-workload-certs\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9\" (UID: \"117032f9-b742-432f-ac5f-70c9982b919f\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" Oct 31 01:13:22.349498 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.349336 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-istio-data\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq\" (UID: \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" Oct 31 01:13:22.349498 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.349372 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f889944d-e872-4b84-baa6-829de12f3ba1-istio-data\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5\" (UID: \"f889944d-e872-4b84-baa6-829de12f3ba1\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" Oct 31 01:13:22.349498 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.349410 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f889944d-e872-4b84-baa6-829de12f3ba1-istio-token\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5\" (UID: \"f889944d-e872-4b84-baa6-829de12f3ba1\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" Oct 31 01:13:22.350127 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.350104 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f889944d-e872-4b84-baa6-829de12f3ba1-workload-socket\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5\" (UID: \"f889944d-e872-4b84-baa6-829de12f3ba1\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" Oct 31 01:13:22.350308 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.350289 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f889944d-e872-4b84-baa6-829de12f3ba1-istio-data\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5\" (UID: \"f889944d-e872-4b84-baa6-829de12f3ba1\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" Oct 31 01:13:22.350954 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.350751 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f889944d-e872-4b84-baa6-829de12f3ba1-workload-certs\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5\" (UID: \"f889944d-e872-4b84-baa6-829de12f3ba1\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" Oct 31 01:13:22.363018 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.362984 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f889944d-e872-4b84-baa6-829de12f3ba1-istio-token\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5\" (UID: \"f889944d-e872-4b84-baa6-829de12f3ba1\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" Oct 31 01:13:22.363180 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.363059 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9586g\" (UniqueName: \"kubernetes.io/projected/f889944d-e872-4b84-baa6-829de12f3ba1-kube-api-access-9586g\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5\" (UID: \"f889944d-e872-4b84-baa6-829de12f3ba1\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" Oct 31 01:13:22.363267 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.363239 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f889944d-e872-4b84-baa6-829de12f3ba1-istio-envoy\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5\" (UID: \"f889944d-e872-4b84-baa6-829de12f3ba1\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" Oct 31 01:13:22.363640 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.363616 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f889944d-e872-4b84-baa6-829de12f3ba1-istio-podinfo\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5\" (UID: \"f889944d-e872-4b84-baa6-829de12f3ba1\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" Oct 31 01:13:22.371611 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.371589 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" Oct 31 01:13:22.440551 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.439689 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58"] Oct 31 01:13:22.450714 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.446935 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" Oct 31 01:13:22.455074 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.451808 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/117032f9-b742-432f-ac5f-70c9982b919f-credential-socket\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9\" (UID: \"117032f9-b742-432f-ac5f-70c9982b919f\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" Oct 31 01:13:22.455074 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.451851 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-istio-envoy\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq\" (UID: \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" Oct 31 01:13:22.455074 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.452638 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/117032f9-b742-432f-ac5f-70c9982b919f-credential-socket\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9\" (UID: \"117032f9-b742-432f-ac5f-70c9982b919f\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" Oct 31 01:13:22.455074 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.452715 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/117032f9-b742-432f-ac5f-70c9982b919f-istio-token\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9\" (UID: \"117032f9-b742-432f-ac5f-70c9982b919f\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" Oct 31 01:13:22.455074 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.452772 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-istiod-ca-cert\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq\" (UID: \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" Oct 31 01:13:22.455074 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.452826 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/117032f9-b742-432f-ac5f-70c9982b919f-istiod-ca-cert\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9\" (UID: \"117032f9-b742-432f-ac5f-70c9982b919f\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" Oct 31 01:13:22.455074 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.452852 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-workload-socket\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq\" (UID: \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" Oct 31 01:13:22.455074 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.452901 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-workload-certs\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq\" (UID: \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" Oct 31 01:13:22.455074 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.452942 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/117032f9-b742-432f-ac5f-70c9982b919f-istio-envoy\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9\" (UID: \"117032f9-b742-432f-ac5f-70c9982b919f\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" Oct 31 01:13:22.455074 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.452964 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/117032f9-b742-432f-ac5f-70c9982b919f-istio-data\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9\" (UID: \"117032f9-b742-432f-ac5f-70c9982b919f\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" Oct 31 01:13:22.455074 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.452994 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-credential-socket\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq\" (UID: \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" Oct 31 01:13:22.455074 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.453029 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-istio-podinfo\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq\" (UID: \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" Oct 31 01:13:22.455074 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.453075 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/117032f9-b742-432f-ac5f-70c9982b919f-workload-socket\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9\" (UID: \"117032f9-b742-432f-ac5f-70c9982b919f\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" Oct 31 01:13:22.455074 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.453106 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8j59\" (UniqueName: \"kubernetes.io/projected/117032f9-b742-432f-ac5f-70c9982b919f-kube-api-access-v8j59\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9\" (UID: \"117032f9-b742-432f-ac5f-70c9982b919f\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" Oct 31 01:13:22.455074 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.453203 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk6vb\" (UniqueName: \"kubernetes.io/projected/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-kube-api-access-bk6vb\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq\" (UID: \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" Oct 31 01:13:22.455074 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.453288 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/117032f9-b742-432f-ac5f-70c9982b919f-istio-podinfo\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9\" (UID: \"117032f9-b742-432f-ac5f-70c9982b919f\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" Oct 31 01:13:22.455074 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.453311 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-istio-token\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq\" (UID: \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" Oct 31 01:13:22.455074 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.453365 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/117032f9-b742-432f-ac5f-70c9982b919f-workload-certs\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9\" (UID: \"117032f9-b742-432f-ac5f-70c9982b919f\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" Oct 31 01:13:22.464480 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.459446 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-credential-socket\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq\" (UID: \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" Oct 31 01:13:22.464480 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.460347 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-istiod-ca-cert\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq\" (UID: \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" Oct 31 01:13:22.464480 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.460932 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/117032f9-b742-432f-ac5f-70c9982b919f-istiod-ca-cert\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9\" (UID: \"117032f9-b742-432f-ac5f-70c9982b919f\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" Oct 31 01:13:22.464480 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.461205 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-workload-socket\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq\" (UID: \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" Oct 31 01:13:22.464480 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.461448 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-workload-certs\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq\" (UID: \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" Oct 31 01:13:22.469976 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.466518 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58"] Oct 31 01:13:22.470246 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.470225 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/117032f9-b742-432f-ac5f-70c9982b919f-istio-token\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9\" (UID: \"117032f9-b742-432f-ac5f-70c9982b919f\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" Oct 31 01:13:22.470867 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.470845 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/117032f9-b742-432f-ac5f-70c9982b919f-istio-data\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9\" (UID: \"117032f9-b742-432f-ac5f-70c9982b919f\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" Oct 31 01:13:22.474134 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.472716 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/117032f9-b742-432f-ac5f-70c9982b919f-workload-socket\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9\" (UID: \"117032f9-b742-432f-ac5f-70c9982b919f\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" Oct 31 01:13:22.474922 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.474446 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/117032f9-b742-432f-ac5f-70c9982b919f-workload-certs\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9\" (UID: \"117032f9-b742-432f-ac5f-70c9982b919f\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" Oct 31 01:13:22.477257 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.475316 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-istio-data\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq\" (UID: \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" Oct 31 01:13:22.486317 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.453409 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-istio-data\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq\" (UID: \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" Oct 31 01:13:22.490862 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.490834 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8j59\" (UniqueName: \"kubernetes.io/projected/117032f9-b742-432f-ac5f-70c9982b919f-kube-api-access-v8j59\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9\" (UID: \"117032f9-b742-432f-ac5f-70c9982b919f\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" Oct 31 01:13:22.497201 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.497169 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/117032f9-b742-432f-ac5f-70c9982b919f-istio-podinfo\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9\" (UID: \"117032f9-b742-432f-ac5f-70c9982b919f\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" Oct 31 01:13:22.497344 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.497260 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk6vb\" (UniqueName: \"kubernetes.io/projected/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-kube-api-access-bk6vb\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq\" (UID: \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" Oct 31 01:13:22.497613 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.497593 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-istio-token\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq\" (UID: \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" Oct 31 01:13:22.508594 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.508256 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/117032f9-b742-432f-ac5f-70c9982b919f-istio-envoy\") pod \"e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9\" (UID: \"117032f9-b742-432f-ac5f-70c9982b919f\") " pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" Oct 31 01:13:22.508594 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.508512 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-istio-envoy\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq\" (UID: \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" Oct 31 01:13:22.508759 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.508724 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-istio-podinfo\") pod \"e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq\" (UID: \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\") " pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" Oct 31 01:13:22.555965 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.555882 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" Oct 31 01:13:22.592526 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.592306 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f420497-f98d-4a0c-8552-fca5c3f002f2-credential-socket\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58\" (UID: \"5f420497-f98d-4a0c-8552-fca5c3f002f2\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" Oct 31 01:13:22.592526 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.592373 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f420497-f98d-4a0c-8552-fca5c3f002f2-istio-envoy\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58\" (UID: \"5f420497-f98d-4a0c-8552-fca5c3f002f2\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" Oct 31 01:13:22.594146 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.592825 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f420497-f98d-4a0c-8552-fca5c3f002f2-workload-certs\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58\" (UID: \"5f420497-f98d-4a0c-8552-fca5c3f002f2\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" Oct 31 01:13:22.594146 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.592865 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsh4t\" (UniqueName: \"kubernetes.io/projected/5f420497-f98d-4a0c-8552-fca5c3f002f2-kube-api-access-tsh4t\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58\" (UID: \"5f420497-f98d-4a0c-8552-fca5c3f002f2\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" Oct 31 01:13:22.594146 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.592892 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f420497-f98d-4a0c-8552-fca5c3f002f2-istio-token\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58\" (UID: \"5f420497-f98d-4a0c-8552-fca5c3f002f2\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" Oct 31 01:13:22.594146 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.593026 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f420497-f98d-4a0c-8552-fca5c3f002f2-workload-socket\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58\" (UID: \"5f420497-f98d-4a0c-8552-fca5c3f002f2\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" Oct 31 01:13:22.594146 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.593091 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f420497-f98d-4a0c-8552-fca5c3f002f2-istio-podinfo\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58\" (UID: \"5f420497-f98d-4a0c-8552-fca5c3f002f2\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" Oct 31 01:13:22.594146 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.593168 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f420497-f98d-4a0c-8552-fca5c3f002f2-istiod-ca-cert\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58\" (UID: \"5f420497-f98d-4a0c-8552-fca5c3f002f2\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" Oct 31 01:13:22.594146 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.593205 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f420497-f98d-4a0c-8552-fca5c3f002f2-istio-data\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58\" (UID: \"5f420497-f98d-4a0c-8552-fca5c3f002f2\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" Oct 31 01:13:22.652778 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.652736 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" Oct 31 01:13:22.662789 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.662526 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" Oct 31 01:13:22.670543 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.670501 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br"] Oct 31 01:13:22.671389 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:13:22.671359 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66b984db_1113_44af_b6d9_6cf452c97187.slice/crio-ae091e58577972c45679de73f8cc337e8ef04d265cf5a3024203d4d68136cf6e WatchSource:0}: Error finding container ae091e58577972c45679de73f8cc337e8ef04d265cf5a3024203d4d68136cf6e: Status 404 returned error can't find the container with id ae091e58577972c45679de73f8cc337e8ef04d265cf5a3024203d4d68136cf6e Oct 31 01:13:22.674627 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.674583 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:13:22.674745 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.674695 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:13:22.674788 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.674749 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:13:22.695490 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.695406 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f420497-f98d-4a0c-8552-fca5c3f002f2-credential-socket\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58\" (UID: \"5f420497-f98d-4a0c-8552-fca5c3f002f2\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" Oct 31 01:13:22.695625 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.695518 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f420497-f98d-4a0c-8552-fca5c3f002f2-istio-envoy\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58\" (UID: \"5f420497-f98d-4a0c-8552-fca5c3f002f2\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" Oct 31 01:13:22.695717 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.695700 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f420497-f98d-4a0c-8552-fca5c3f002f2-workload-certs\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58\" (UID: \"5f420497-f98d-4a0c-8552-fca5c3f002f2\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" Oct 31 01:13:22.695776 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.695733 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsh4t\" (UniqueName: \"kubernetes.io/projected/5f420497-f98d-4a0c-8552-fca5c3f002f2-kube-api-access-tsh4t\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58\" (UID: \"5f420497-f98d-4a0c-8552-fca5c3f002f2\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" Oct 31 01:13:22.695776 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.695764 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f420497-f98d-4a0c-8552-fca5c3f002f2-istio-token\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58\" (UID: \"5f420497-f98d-4a0c-8552-fca5c3f002f2\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" Oct 31 01:13:22.695918 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.695903 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f420497-f98d-4a0c-8552-fca5c3f002f2-workload-socket\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58\" (UID: \"5f420497-f98d-4a0c-8552-fca5c3f002f2\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" Oct 31 01:13:22.695979 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.695963 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f420497-f98d-4a0c-8552-fca5c3f002f2-istio-podinfo\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58\" (UID: \"5f420497-f98d-4a0c-8552-fca5c3f002f2\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" Oct 31 01:13:22.696046 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.696032 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f420497-f98d-4a0c-8552-fca5c3f002f2-istiod-ca-cert\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58\" (UID: \"5f420497-f98d-4a0c-8552-fca5c3f002f2\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" Oct 31 01:13:22.696107 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.696071 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f420497-f98d-4a0c-8552-fca5c3f002f2-istio-data\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58\" (UID: \"5f420497-f98d-4a0c-8552-fca5c3f002f2\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" Oct 31 01:13:22.696624 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.696595 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f420497-f98d-4a0c-8552-fca5c3f002f2-istio-data\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58\" (UID: \"5f420497-f98d-4a0c-8552-fca5c3f002f2\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" Oct 31 01:13:22.696869 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.696846 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f420497-f98d-4a0c-8552-fca5c3f002f2-credential-socket\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58\" (UID: \"5f420497-f98d-4a0c-8552-fca5c3f002f2\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" Oct 31 01:13:22.698169 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.697851 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f420497-f98d-4a0c-8552-fca5c3f002f2-workload-socket\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58\" (UID: \"5f420497-f98d-4a0c-8552-fca5c3f002f2\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" Oct 31 01:13:22.698576 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.698550 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f420497-f98d-4a0c-8552-fca5c3f002f2-workload-certs\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58\" (UID: \"5f420497-f98d-4a0c-8552-fca5c3f002f2\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" Oct 31 01:13:22.698729 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.698574 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f420497-f98d-4a0c-8552-fca5c3f002f2-istiod-ca-cert\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58\" (UID: \"5f420497-f98d-4a0c-8552-fca5c3f002f2\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" Oct 31 01:13:22.699751 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.699723 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f420497-f98d-4a0c-8552-fca5c3f002f2-istio-envoy\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58\" (UID: \"5f420497-f98d-4a0c-8552-fca5c3f002f2\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" Oct 31 01:13:22.704160 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.704110 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f420497-f98d-4a0c-8552-fca5c3f002f2-istio-podinfo\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58\" (UID: \"5f420497-f98d-4a0c-8552-fca5c3f002f2\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" Oct 31 01:13:22.707285 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.707232 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f420497-f98d-4a0c-8552-fca5c3f002f2-istio-token\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58\" (UID: \"5f420497-f98d-4a0c-8552-fca5c3f002f2\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" Oct 31 01:13:22.709133 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.709086 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsh4t\" (UniqueName: \"kubernetes.io/projected/5f420497-f98d-4a0c-8552-fca5c3f002f2-kube-api-access-tsh4t\") pod \"e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58\" (UID: \"5f420497-f98d-4a0c-8552-fca5c3f002f2\") " pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" Oct 31 01:13:22.801318 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.801119 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" Oct 31 01:13:22.882372 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.877144 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:13:22.882372 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.877241 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:13:22.882372 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.877287 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:13:22.882372 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.879896 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5"] Oct 31 01:13:22.979614 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.979570 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq"] Oct 31 01:13:22.983692 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:13:22.983660 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb7394c_0c4a_43e6_bbca_f4222bec42ff.slice/crio-ada5233c87952520037e85e3e499fd0e8280c35822c9ec8a84d2c596913a32bb WatchSource:0}: Error finding container ada5233c87952520037e85e3e499fd0e8280c35822c9ec8a84d2c596913a32bb: Status 404 returned error can't find the container with id ada5233c87952520037e85e3e499fd0e8280c35822c9ec8a84d2c596913a32bb Oct 31 01:13:22.989093 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.988310 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:13:22.989093 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.988399 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:13:22.989093 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:22.988457 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:13:23.017093 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:23.017064 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9"] Oct 31 01:13:23.028106 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:23.028059 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:13:23.028337 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:23.028310 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:13:23.028504 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:23.028484 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:13:23.101019 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:23.100981 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:23.146077 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:23.146044 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58"] Oct 31 01:13:23.173799 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:23.173745 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:13:23.173955 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:23.173848 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:13:23.173955 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:23.173923 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:13:23.653527 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:23.653252 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" event={"ID":"66b984db-1113-44af-b6d9-6cf452c97187","Type":"ContainerStarted","Data":"8283803af9dca21d4b672094bf290e6bab2c020f091996b1a51930df9bdb16aa"} Oct 31 01:13:23.653527 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:23.653337 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" event={"ID":"66b984db-1113-44af-b6d9-6cf452c97187","Type":"ContainerStarted","Data":"ae091e58577972c45679de73f8cc337e8ef04d265cf5a3024203d4d68136cf6e"} Oct 31 01:13:23.656367 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:23.656328 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" event={"ID":"eeb7394c-0c4a-43e6-bbca-f4222bec42ff","Type":"ContainerStarted","Data":"ce2378e90758e17c8ae084350a07551e3ccb018ee1faddd45d9b880089b8a937"} Oct 31 01:13:23.656673 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:23.656641 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" event={"ID":"eeb7394c-0c4a-43e6-bbca-f4222bec42ff","Type":"ContainerStarted","Data":"ada5233c87952520037e85e3e499fd0e8280c35822c9ec8a84d2c596913a32bb"} Oct 31 01:13:23.662092 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:23.662059 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" event={"ID":"5f420497-f98d-4a0c-8552-fca5c3f002f2","Type":"ContainerStarted","Data":"43a4bbc42c552d99a20b0ed441a4ece0bbaab3bcb4f7a2cfa71b6d4f1e76af2c"} Oct 31 01:13:23.662324 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:23.662303 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" event={"ID":"5f420497-f98d-4a0c-8552-fca5c3f002f2","Type":"ContainerStarted","Data":"d3e9828f5aa49a3f4215f928b605946910bfef359c08da2d5eb380358d3c8b9e"} Oct 31 01:13:23.665997 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:23.665902 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" event={"ID":"f889944d-e872-4b84-baa6-829de12f3ba1","Type":"ContainerStarted","Data":"7e9c4a7ae854dee3a08c0a09f9e8767db889c6b01e7efb79faa5c8464e615674"} Oct 31 01:13:23.665997 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:23.665963 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" event={"ID":"f889944d-e872-4b84-baa6-829de12f3ba1","Type":"ContainerStarted","Data":"3c9f1466909b2395696b1b564295edc51efb85d3a8b6a42cca344c7d26562884"} Oct 31 01:13:23.668651 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:23.668613 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" event={"ID":"117032f9-b742-432f-ac5f-70c9982b919f","Type":"ContainerStarted","Data":"47d075176e3198c8e4bdd83ab0bdc4c02916e8a58f7aa8ecbe149e38243074af"} Oct 31 01:13:23.668806 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:23.668674 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" event={"ID":"117032f9-b742-432f-ac5f-70c9982b919f","Type":"ContainerStarted","Data":"7f2a94b35e2c86ba3b8a4b22d1c242cacd7ecd2a78d02b8a3a5b9440b9e17027"} Oct 31 01:13:23.674619 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:23.674548 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" podStartSLOduration=1.674527197 podStartE2EDuration="1.674527197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:23.672511019 +0000 UTC m=+1048.031233338" watchObservedRunningTime="2026-10-31 01:13:23.674527197 +0000 UTC m=+1048.033249522" Oct 31 01:13:23.702629 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:23.702488 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" podStartSLOduration=1.702462084 podStartE2EDuration="1.702462084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:23.694440263 +0000 UTC m=+1048.053162573" watchObservedRunningTime="2026-10-31 01:13:23.702462084 +0000 UTC m=+1048.061184405" Oct 31 01:13:23.763730 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:23.758624 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" podStartSLOduration=1.758600776 podStartE2EDuration="1.758600776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:23.734014449 +0000 UTC m=+1048.092736766" watchObservedRunningTime="2026-10-31 01:13:23.758600776 +0000 UTC m=+1048.117323093" Oct 31 01:13:23.794502 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:23.787018 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" podStartSLOduration=1.786986464 podStartE2EDuration="1.786986464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:23.774144949 +0000 UTC m=+1048.132867266" watchObservedRunningTime="2026-10-31 01:13:23.786986464 +0000 UTC m=+1048.145708782" Oct 31 01:13:23.829017 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:23.828936 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" podStartSLOduration=1.828909831 podStartE2EDuration="1.828909831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:23.813034301 +0000 UTC m=+1048.171756619" watchObservedRunningTime="2026-10-31 01:13:23.828909831 +0000 UTC m=+1048.187632150" Oct 31 01:13:23.858679 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:23.858646 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" Oct 31 01:13:24.272814 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:24.272775 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:24.373831 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:24.373800 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" Oct 31 01:13:24.384610 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:24.384572 4440 patch_prober.go:29] interesting pod/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 31 01:13:24.384874 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:24.384844 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" podUID="66b984db-1113-44af-b6d9-6cf452c97187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 31 01:13:24.564102 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:24.561750 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" Oct 31 01:13:24.569724 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:24.569673 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 31 01:13:24.570028 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:24.569988 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" podUID="f889944d-e872-4b84-baa6-829de12f3ba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 31 01:13:24.657539 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:24.657499 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" Oct 31 01:13:24.658210 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:24.658183 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 01:13:24.658390 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:24.658367 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" podUID="eeb7394c-0c4a-43e6-bbca-f4222bec42ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 01:13:24.675956 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:24.675862 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" Oct 31 01:13:24.742072 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:24.742032 4440 patch_prober.go:29] interesting pod/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 01:13:24.742368 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:24.742339 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" podUID="117032f9-b742-432f-ac5f-70c9982b919f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 01:13:24.819531 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:24.814250 4440 patch_prober.go:29] interesting pod/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 31 01:13:24.819531 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:24.814321 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" podUID="5f420497-f98d-4a0c-8552-fca5c3f002f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 31 01:13:25.032259 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:25.032191 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:25.058628 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:25.058577 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:25.078635 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:25.078525 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 31 01:13:25.258617 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:25.258561 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:25.373042 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:25.372938 4440 patch_prober.go:29] interesting pod/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 31 01:13:25.373307 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:25.373274 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" podUID="66b984db-1113-44af-b6d9-6cf452c97187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 31 01:13:25.539324 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:25.539274 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:25.564357 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:25.557004 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 31 01:13:25.568985 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:25.568546 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" podUID="f889944d-e872-4b84-baa6-829de12f3ba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 31 01:13:25.656772 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:25.656663 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" podUID="0b82b50e-bfa7-45b9-8484-c7a331b5ac03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:25.659301 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:25.659271 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 01:13:25.659970 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:25.659941 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" podUID="eeb7394c-0c4a-43e6-bbca-f4222bec42ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 01:13:25.668767 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:25.668559 4440 patch_prober.go:29] interesting pod/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 01:13:25.668767 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:25.668616 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" podUID="117032f9-b742-432f-ac5f-70c9982b919f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 01:13:25.860482 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:25.835301 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" podUID="617e420d-d029-4f4b-8ae5-63fe7d85971e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:25.860482 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:25.838638 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" podUID="cf86081d-7942-454a-8763-f658bd0ce870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:25.860482 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:25.838730 4440 patch_prober.go:29] interesting pod/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 31 01:13:25.860482 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:25.857353 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" podUID="5f420497-f98d-4a0c-8552-fca5c3f002f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 31 01:13:25.894528 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:25.892532 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:25.917509 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:25.902758 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" podUID="2f72d419-0877-4e49-9a85-095ddea8c790" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:26.011826 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:26.011780 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:26.014925 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:26.012091 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 31 01:13:26.035453 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:26.034525 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:26.035453 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:26.035370 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" Oct 31 01:13:26.071088 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:26.071035 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:26.072191 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:26.072157 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" Oct 31 01:13:26.105494 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:26.086512 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:26.105494 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:26.086920 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:26.105494 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:26.087563 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" Oct 31 01:13:26.236355 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:26.234560 4440 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:26.283353 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:26.259628 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:26.283353 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:26.260710 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" Oct 31 01:13:26.437452 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:26.407463 4440 patch_prober.go:29] interesting pod/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 31 01:13:26.437452 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:26.407548 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" podUID="66b984db-1113-44af-b6d9-6cf452c97187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 31 01:13:26.558020 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:26.557927 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 31 01:13:26.558273 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:26.558243 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" podUID="f889944d-e872-4b84-baa6-829de12f3ba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 31 01:13:26.660448 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:26.660126 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" podUID="0b82b50e-bfa7-45b9-8484-c7a331b5ac03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:26.660997 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:26.660807 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" Oct 31 01:13:26.860502 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:26.860222 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" podUID="cf86081d-7942-454a-8763-f658bd0ce870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:26.944509 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:26.881673 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" Oct 31 01:13:26.944509 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:26.882390 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" podUID="617e420d-d029-4f4b-8ae5-63fe7d85971e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:26.944509 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:26.887567 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" Oct 31 01:13:26.944509 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:26.894527 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 31 01:13:26.944509 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:26.895589 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" Oct 31 01:13:26.944509 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:26.909157 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" podUID="2f72d419-0877-4e49-9a85-095ddea8c790" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:26.944509 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:26.915264 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" Oct 31 01:13:26.944509 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:26.924594 4440 patch_prober.go:29] interesting pod/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 31 01:13:26.944509 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:26.924664 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" podUID="5f420497-f98d-4a0c-8552-fca5c3f002f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 31 01:13:26.944509 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:26.924746 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 01:13:26.944509 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:26.924772 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" podUID="eeb7394c-0c4a-43e6-bbca-f4222bec42ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 01:13:26.964329 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:26.952565 4440 patch_prober.go:29] interesting pod/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 01:13:26.964329 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:26.952620 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" podUID="117032f9-b742-432f-ac5f-70c9982b919f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 01:13:27.056685 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:27.048284 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:27.056685 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:27.048584 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:27.056685 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:27.049391 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:27.056685 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:27.049643 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" Oct 31 01:13:27.056685 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:27.049699 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" Oct 31 01:13:27.114508 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:27.083537 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:27.172445 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:27.168534 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 31 01:13:27.223529 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:27.192512 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:27.252803 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:27.224133 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" Oct 31 01:13:27.296444 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:27.254943 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:27.327452 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:27.276651 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 01:13:27.564449 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:27.466982 4440 patch_prober.go:29] interesting pod/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 31 01:13:27.564449 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:27.467046 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" podUID="66b984db-1113-44af-b6d9-6cf452c97187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 31 01:13:27.620931 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:27.594759 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 31 01:13:27.620931 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:27.594834 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" podUID="f889944d-e872-4b84-baa6-829de12f3ba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 31 01:13:27.726443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:27.701113 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 01:13:27.726443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:27.701173 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" podUID="eeb7394c-0c4a-43e6-bbca-f4222bec42ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 01:13:27.726443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:27.703739 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" podUID="0b82b50e-bfa7-45b9-8484-c7a331b5ac03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:27.973514 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:27.911112 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" podUID="2f72d419-0877-4e49-9a85-095ddea8c790" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:27.973514 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:27.971708 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" podUID="cf86081d-7942-454a-8763-f658bd0ce870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 31 01:13:27.977757 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:27.977714 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:28.087461 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:28.076790 4440 patch_prober.go:29] interesting pod/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 31 01:13:28.087461 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:28.076867 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" podUID="5f420497-f98d-4a0c-8552-fca5c3f002f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 31 01:13:28.115444 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:28.109157 4440 patch_prober.go:29] interesting pod/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 01:13:28.115444 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:28.109245 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" podUID="117032f9-b742-432f-ac5f-70c9982b919f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 01:13:28.161868 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:28.120528 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" podUID="617e420d-d029-4f4b-8ae5-63fe7d85971e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:28.237542 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:28.212395 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 31 01:13:28.238233 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:28.238189 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 31 01:13:28.279523 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:28.275551 4440 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" podUID="0b82b50e-bfa7-45b9-8484-c7a331b5ac03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": context deadline exceeded" Oct 31 01:13:28.329530 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:28.297912 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 01:13:28.340479 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:28.338645 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 31 01:13:28.360974 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:28.323818 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 31 01:13:28.360974 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:28.323775 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:28.419656 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:28.419597 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:28.470341 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:28.431570 4440 patch_prober.go:29] interesting pod/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 31 01:13:28.470726 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:28.470685 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" podUID="66b984db-1113-44af-b6d9-6cf452c97187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 31 01:13:28.610502 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:28.568588 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 31 01:13:28.610502 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:28.568656 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" podUID="f889944d-e872-4b84-baa6-829de12f3ba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 31 01:13:28.610502 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:28.568742 4440 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:28.708447 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:28.704693 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" podUID="0b82b50e-bfa7-45b9-8484-c7a331b5ac03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 31 01:13:28.804146 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:28.803029 4440 patch_prober.go:29] interesting pod/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 01:13:28.804146 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:28.803096 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" podUID="117032f9-b742-432f-ac5f-70c9982b919f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 01:13:28.804146 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:28.803191 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 01:13:28.804146 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:28.803215 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" podUID="eeb7394c-0c4a-43e6-bbca-f4222bec42ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 01:13:28.804146 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:28.804039 4440 patch_prober.go:29] interesting pod/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 31 01:13:28.804146 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:28.804082 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" podUID="5f420497-f98d-4a0c-8552-fca5c3f002f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 31 01:13:29.035453 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:29.031747 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" podUID="cf86081d-7942-454a-8763-f658bd0ce870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 31 01:13:29.035453 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:29.032181 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" podUID="2f72d419-0877-4e49-9a85-095ddea8c790" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Oct 31 01:13:29.054904 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:29.054847 4440 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" podUID="2f72d419-0877-4e49-9a85-095ddea8c790" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:29.153905 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:29.135404 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" podUID="617e420d-d029-4f4b-8ae5-63fe7d85971e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:29.153905 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:29.138785 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:29.269586 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:29.247515 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:29.269586 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:29.258559 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 31 01:13:29.269586 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:29.268142 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:29.367562 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:29.346524 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 01:13:29.369509 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:29.369461 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:29.381640 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:29.372745 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:29.381640 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:29.378693 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 01:13:29.507444 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:29.404846 4440 patch_prober.go:29] interesting pod/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 31 01:13:29.507444 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:29.404921 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" podUID="66b984db-1113-44af-b6d9-6cf452c97187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 31 01:13:29.507444 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:29.479810 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 31 01:13:29.591605 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:29.579354 4440 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:29.591605 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:29.591060 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 31 01:13:29.591605 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:29.591131 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" podUID="f889944d-e872-4b84-baa6-829de12f3ba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 31 01:13:29.668048 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:29.668005 4440 patch_prober.go:29] interesting pod/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 01:13:29.717295 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:29.717252 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" podUID="117032f9-b742-432f-ac5f-70c9982b919f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 01:13:29.717612 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:29.716122 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 01:13:29.726514 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:29.721661 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" podUID="0b82b50e-bfa7-45b9-8484-c7a331b5ac03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:29.752345 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:29.752169 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" podUID="eeb7394c-0c4a-43e6-bbca-f4222bec42ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 01:13:29.811031 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:29.810981 4440 patch_prober.go:29] interesting pod/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 31 01:13:29.899695 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:29.899639 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" podUID="5f420497-f98d-4a0c-8552-fca5c3f002f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 31 01:13:30.034957 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:30.034853 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" podUID="2f72d419-0877-4e49-9a85-095ddea8c790" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:30.035570 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:30.035537 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" podUID="cf86081d-7942-454a-8763-f658bd0ce870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 31 01:13:30.141394 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:30.141338 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" podUID="617e420d-d029-4f4b-8ae5-63fe7d85971e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:30.279449 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:30.248965 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:30.280790 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:30.280745 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 31 01:13:30.313643 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:30.312830 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:30.347601 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:30.347223 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:30.373523 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:30.373478 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:30.374135 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:30.374110 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 31 01:13:30.385780 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:30.382563 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 01:13:30.421966 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:30.420507 4440 patch_prober.go:29] interesting pod/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 31 01:13:30.421966 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:30.420575 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" podUID="66b984db-1113-44af-b6d9-6cf452c97187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 31 01:13:30.486589 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:30.486210 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 31 01:13:30.578958 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:30.578861 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 31 01:13:30.604445 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:30.579196 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" podUID="f889944d-e872-4b84-baa6-829de12f3ba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 31 01:13:30.656446 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:30.654455 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 01:13:30.656446 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:30.654526 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" podUID="eeb7394c-0c4a-43e6-bbca-f4222bec42ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 01:13:30.671469 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:30.670503 4440 patch_prober.go:29] interesting pod/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 01:13:30.671469 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:30.670565 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" podUID="117032f9-b742-432f-ac5f-70c9982b919f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 01:13:30.745158 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:30.745107 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" podUID="0b82b50e-bfa7-45b9-8484-c7a331b5ac03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:30.802893 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:30.802851 4440 patch_prober.go:29] interesting pod/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 31 01:13:30.803234 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:30.803200 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" podUID="5f420497-f98d-4a0c-8552-fca5c3f002f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 31 01:13:31.058444 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:31.041594 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" podUID="2f72d419-0877-4e49-9a85-095ddea8c790" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Oct 31 01:13:31.058444 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:31.042080 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" podUID="cf86081d-7942-454a-8763-f658bd0ce870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:31.094100 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:31.093515 4440 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:31.148449 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:31.147541 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" podUID="617e420d-d029-4f4b-8ae5-63fe7d85971e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:31.194177 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:31.194123 4440 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:31.260328 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:31.260282 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:31.313508 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:31.312521 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:31.313508 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:31.312689 4440 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:31.318596 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:31.313840 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:31.334529 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:13:31.334441 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:13:31.335268 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:31.335224 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:31.351879 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:31.350540 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 01:13:31.385567 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:31.382523 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:31.385567 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:31.382937 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 31 01:13:31.386229 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:31.386198 4440 patch_prober.go:29] interesting pod/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 31 01:13:31.386470 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:31.386393 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" podUID="66b984db-1113-44af-b6d9-6cf452c97187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 31 01:13:31.386767 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:31.386677 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:31.490608 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:31.490559 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:31.570501 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:31.570090 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 31 01:13:31.570501 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:31.570165 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" podUID="f889944d-e872-4b84-baa6-829de12f3ba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 31 01:13:31.671899 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:31.671859 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 01:13:31.672190 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:31.672153 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" podUID="eeb7394c-0c4a-43e6-bbca-f4222bec42ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 01:13:31.672399 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:31.672381 4440 patch_prober.go:29] interesting pod/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 01:13:31.672551 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:31.672530 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" podUID="117032f9-b742-432f-ac5f-70c9982b919f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 01:13:31.775544 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:31.772336 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" podUID="0b82b50e-bfa7-45b9-8484-c7a331b5ac03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:31.859501 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:31.821499 4440 patch_prober.go:29] interesting pod/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 31 01:13:31.861091 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:31.861044 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" podUID="5f420497-f98d-4a0c-8552-fca5c3f002f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 31 01:13:32.095450 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:32.091514 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" podUID="cf86081d-7942-454a-8763-f658bd0ce870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:32.163541 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:32.148615 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" podUID="617e420d-d029-4f4b-8ae5-63fe7d85971e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:32.165157 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:32.102625 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" podUID="2f72d419-0877-4e49-9a85-095ddea8c790" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:32.268459 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:32.262538 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:32.326828 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:32.319529 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:32.326828 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:32.322231 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:32.355911 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:32.354122 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:32.406178 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:32.399867 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:32.406563 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:32.406522 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:32.407892 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:32.407867 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 31 01:13:32.431462 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:32.430401 4440 patch_prober.go:29] interesting pod/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 31 01:13:32.431462 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:32.430489 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" podUID="66b984db-1113-44af-b6d9-6cf452c97187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 31 01:13:32.501857 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:32.494551 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:32.529936 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:32.513198 4440 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" podUID="cf86081d-7942-454a-8763-f658bd0ce870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:32.635185 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:32.635142 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 31 01:13:32.635520 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:32.635475 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" podUID="f889944d-e872-4b84-baa6-829de12f3ba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 31 01:13:32.676447 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:32.676273 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 01:13:32.676447 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:32.676332 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" podUID="eeb7394c-0c4a-43e6-bbca-f4222bec42ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 01:13:32.677335 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:32.677230 4440 patch_prober.go:29] interesting pod/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 01:13:32.677335 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:32.677278 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" podUID="117032f9-b742-432f-ac5f-70c9982b919f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 01:13:32.710592 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:32.708724 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:32.780547 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:32.775504 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" podUID="0b82b50e-bfa7-45b9-8484-c7a331b5ac03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 31 01:13:32.808533 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:32.808084 4440 patch_prober.go:29] interesting pod/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 31 01:13:32.808533 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:32.808155 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" podUID="5f420497-f98d-4a0c-8552-fca5c3f002f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 31 01:13:33.150716 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:33.150580 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" podUID="cf86081d-7942-454a-8763-f658bd0ce870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:33.165444 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:33.164639 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" podUID="617e420d-d029-4f4b-8ae5-63fe7d85971e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:33.165663 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:33.165497 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" podUID="2f72d419-0877-4e49-9a85-095ddea8c790" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:33.264631 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:33.264570 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:33.346443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:33.322586 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 31 01:13:33.346443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:33.323058 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 31 01:13:33.370444 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:33.354554 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:33.424499 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:33.417110 4440 patch_prober.go:29] interesting pod/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 31 01:13:33.424499 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:33.417171 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" podUID="66b984db-1113-44af-b6d9-6cf452c97187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 31 01:13:33.424499 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:33.417994 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:33.424499 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:33.418377 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:33.424499 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:33.418657 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 31 01:13:33.440440 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:33.427135 4440 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": context deadline exceeded" Oct 31 01:13:33.496375 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:33.496326 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 31 01:13:33.557216 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:33.557172 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 31 01:13:33.557512 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:33.557484 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" podUID="f889944d-e872-4b84-baa6-829de12f3ba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 31 01:13:33.648916 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:33.648805 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:33.682699 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:33.682582 4440 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" podUID="617e420d-d029-4f4b-8ae5-63fe7d85971e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:33.918965 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:33.683300 4440 patch_prober.go:29] interesting pod/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 01:13:33.961514 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:33.958565 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" podUID="117032f9-b742-432f-ac5f-70c9982b919f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 01:13:33.961514 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:33.778577 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" podUID="0b82b50e-bfa7-45b9-8484-c7a331b5ac03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:33.961514 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:33.683596 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 01:13:33.961514 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:33.959484 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" podUID="eeb7394c-0c4a-43e6-bbca-f4222bec42ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 01:13:34.056157 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:34.056084 4440 patch_prober.go:29] interesting pod/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 31 01:13:34.085444 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:34.084530 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" podUID="5f420497-f98d-4a0c-8552-fca5c3f002f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 31 01:13:34.109753 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:34.109700 4440 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:34.156376 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:34.156332 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" podUID="cf86081d-7942-454a-8763-f658bd0ce870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 31 01:13:34.174670 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:34.174607 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" podUID="617e420d-d029-4f4b-8ae5-63fe7d85971e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": context deadline exceeded" Oct 31 01:13:34.175494 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:34.175453 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" podUID="2f72d419-0877-4e49-9a85-095ddea8c790" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:34.285557 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:34.279310 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 31 01:13:34.359444 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:34.333638 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 31 01:13:34.359444 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:34.334069 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 31 01:13:34.359444 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:34.357773 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:34.403443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:34.372205 4440 patch_prober.go:29] interesting pod/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 31 01:13:34.412455 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:34.403729 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" podUID="66b984db-1113-44af-b6d9-6cf452c97187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 31 01:13:34.466484 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:34.434719 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:34.568001 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:34.436044 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 01:13:34.614035 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:34.436089 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:34.618882 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:34.525516 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 31 01:13:34.620059 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:34.566630 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 31 01:13:34.639440 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:34.634542 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" podUID="f889944d-e872-4b84-baa6-829de12f3ba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 31 01:13:34.639440 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:34.584431 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:34.670408 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:34.670366 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 01:13:34.693575 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:34.693521 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" podUID="eeb7394c-0c4a-43e6-bbca-f4222bec42ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 01:13:34.693870 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:34.688435 4440 patch_prober.go:29] interesting pod/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 01:13:34.694030 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:34.694003 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" podUID="117032f9-b742-432f-ac5f-70c9982b919f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 01:13:34.765448 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:34.765391 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" Oct 31 01:13:34.786593 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:34.786555 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" Oct 31 01:13:34.806979 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:34.806943 4440 patch_prober.go:29] interesting pod/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 31 01:13:34.809551 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:34.809510 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" podUID="5f420497-f98d-4a0c-8552-fca5c3f002f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 31 01:13:34.987505 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:34.921330 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" Oct 31 01:13:35.032443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:35.031810 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" Oct 31 01:13:35.164128 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:35.164064 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" podUID="cf86081d-7942-454a-8763-f658bd0ce870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:35.188453 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:35.185577 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" podUID="2f72d419-0877-4e49-9a85-095ddea8c790" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:35.191106 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:35.191062 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" podUID="617e420d-d029-4f4b-8ae5-63fe7d85971e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:35.302526 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:35.295533 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:35.380781 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:35.380744 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:35.393775 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:35.393739 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" Oct 31 01:13:35.413690 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:35.413628 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 31 01:13:35.439445 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:35.413978 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 31 01:13:35.633644 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:35.633539 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:35.662445 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:35.652607 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:35.662445 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:35.654733 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 01:13:35.662445 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:35.654784 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" podUID="eeb7394c-0c4a-43e6-bbca-f4222bec42ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 01:13:35.685955 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:35.685918 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 31 01:13:35.686307 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:35.686254 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" podUID="f889944d-e872-4b84-baa6-829de12f3ba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 31 01:13:35.708871 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:35.704495 4440 patch_prober.go:29] interesting pod/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 01:13:35.708871 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:35.704561 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" podUID="117032f9-b742-432f-ac5f-70c9982b919f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 01:13:35.843622 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:35.843583 4440 patch_prober.go:29] interesting pod/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 31 01:13:35.843882 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:35.843851 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" podUID="5f420497-f98d-4a0c-8552-fca5c3f002f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 31 01:13:36.170014 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:36.166553 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" podUID="cf86081d-7942-454a-8763-f658bd0ce870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 31 01:13:36.206236 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:36.206174 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" podUID="2f72d419-0877-4e49-9a85-095ddea8c790" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:36.232586 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:36.232544 4440 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:36.269498 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:36.247511 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" Oct 31 01:13:36.374638 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:36.374574 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:36.453855 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:36.398074 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 01:13:36.517984 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:36.517741 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 01:13:36.518195 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:36.518150 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 31 01:13:36.693452 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:36.692396 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 31 01:13:36.784991 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:36.784869 4440 patch_prober.go:29] interesting pod/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:13:36.785288 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:36.785238 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" podUID="66b984db-1113-44af-b6d9-6cf452c97187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:37.088615 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:37.088507 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" podUID="0b82b50e-bfa7-45b9-8484-c7a331b5ac03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:37.204679 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:37.204635 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:37.207628 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:37.205272 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:37.248746 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:37.248702 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" podUID="2f72d419-0877-4e49-9a85-095ddea8c790" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:37.269671 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:37.269630 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" podUID="cf86081d-7942-454a-8763-f658bd0ce870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:37.270508 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:37.270460 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" Oct 31 01:13:37.296443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:37.276596 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" Oct 31 01:13:37.296443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:37.292878 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" Oct 31 01:13:37.300049 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:37.300018 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:37.306108 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:37.305903 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" podUID="617e420d-d029-4f4b-8ae5-63fe7d85971e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:37.350958 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:37.350870 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" Oct 31 01:13:37.486012 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:37.484222 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" Oct 31 01:13:37.486012 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:37.484314 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:13:37.495483 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:37.486693 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" Oct 31 01:13:37.495483 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:37.486863 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" Oct 31 01:13:37.495483 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:37.486961 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" Oct 31 01:13:37.495483 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:37.487052 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" Oct 31 01:13:37.495483 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:37.488948 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" Oct 31 01:13:37.575882 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:37.558632 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" Oct 31 01:13:37.575882 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:37.558763 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" Oct 31 01:13:37.575882 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:37.558834 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" Oct 31 01:13:37.575882 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:37.558913 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" Oct 31 01:13:37.575882 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:37.564007 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" Oct 31 01:13:37.575882 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:37.564070 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" Oct 31 01:13:37.575882 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:37.567163 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" Oct 31 01:13:37.575882 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:37.568549 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" Oct 31 01:13:37.575882 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:37.569168 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" Oct 31 01:13:37.575882 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:37.569280 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" Oct 31 01:13:37.575882 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:37.569856 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:37.575882 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:37.574283 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" Oct 31 01:13:37.916552 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:37.916517 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:39.173773 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:39.173532 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:39.246885 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:39.246560 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:39.272446 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:39.258586 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:39.333442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:39.332745 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:39.357445 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:39.347547 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:39.377445 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:39.365304 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:39.567627 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:39.567521 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 31 01:13:39.658575 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:39.658518 4440 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:39.757594 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:39.757535 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" podUID="0b82b50e-bfa7-45b9-8484-c7a331b5ac03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:39.946443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:39.942524 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" podUID="2f72d419-0877-4e49-9a85-095ddea8c790" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:40.043582 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:40.043520 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" podUID="cf86081d-7942-454a-8763-f658bd0ce870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:40.061453 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:40.058786 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:40.126957 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:40.126898 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" podUID="617e420d-d029-4f4b-8ae5-63fe7d85971e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": context deadline exceeded" Oct 31 01:13:40.184013 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:40.183966 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 31 01:13:40.185656 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:40.185623 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" Oct 31 01:13:40.320605 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:40.320492 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 31 01:13:40.602177 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:40.369307 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 31 01:13:40.662502 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:40.370056 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 31 01:13:40.662502 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:40.631050 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" Oct 31 01:13:40.662502 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:40.631170 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" Oct 31 01:13:40.662502 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:40.631243 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" Oct 31 01:13:40.662502 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:40.370106 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:40.662502 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:40.370129 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 01:13:40.673784 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:40.581953 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:40.677093 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:40.677062 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" Oct 31 01:13:40.677406 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:40.677390 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" Oct 31 01:13:40.677838 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:40.677823 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" Oct 31 01:13:40.820903 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:40.820849 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" podUID="0b82b50e-bfa7-45b9-8484-c7a331b5ac03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:40.823251 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:40.823215 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" Oct 31 01:13:41.062590 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:41.062537 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" podUID="cf86081d-7942-454a-8763-f658bd0ce870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:41.064128 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:41.064102 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" Oct 31 01:13:41.090984 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:41.070570 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 31 01:13:41.090984 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:41.079591 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" Oct 31 01:13:41.117724 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:41.117671 4440 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:41.235445 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:41.130374 4440 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": context deadline exceeded" Oct 31 01:13:41.334929 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:41.334821 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" podUID="617e420d-d029-4f4b-8ae5-63fe7d85971e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:41.356961 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:41.356925 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" Oct 31 01:13:41.370445 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:41.359955 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 31 01:13:41.658115 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:41.390740 4440 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:41.658115 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:41.454779 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" podUID="2f72d419-0877-4e49-9a85-095ddea8c790" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Oct 31 01:13:41.658115 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:41.603382 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:41.658115 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:41.656907 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" Oct 31 01:13:41.658972 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:41.658511 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:41.663616 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:41.609681 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 31 01:13:41.712491 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:41.688334 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:41.754083 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:41.754034 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:41.755879 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:41.752636 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 01:13:42.018571 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:41.756233 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:42.069457 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:42.061488 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tvltr" podUID="c14a3342-f1ee-4856-97ff-4c0cc1e25e19" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:42.147441 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:42.139173 4440 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kr4hj" podUID="76e1f133-7f40-4472-a3f3-c1b6944818d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": context deadline exceeded" Oct 31 01:13:42.184744 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:42.164512 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:42.242559 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:42.242505 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" podUID="cf86081d-7942-454a-8763-f658bd0ce870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 31 01:13:42.246510 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:41.887638 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" podUID="0b82b50e-bfa7-45b9-8484-c7a331b5ac03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 31 01:13:42.444227 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:42.444189 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" podUID="617e420d-d029-4f4b-8ae5-63fe7d85971e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": context deadline exceeded" Oct 31 01:13:42.587445 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:42.572549 4440 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" podUID="cf86081d-7942-454a-8763-f658bd0ce870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:42.680660 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:42.680585 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:42.750495 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:42.747214 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:43.082513 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:43.063646 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:43.086463 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:43.084107 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:43.105749 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:43.101519 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tvltr" podUID="c14a3342-f1ee-4856-97ff-4c0cc1e25e19" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:43.190830 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:43.188934 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 31 01:13:43.215665 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:43.191125 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" podUID="2f72d419-0877-4e49-9a85-095ddea8c790" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Oct 31 01:13:43.217181 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:43.215980 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 31 01:13:43.262496 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:43.248667 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" podUID="cf86081d-7942-454a-8763-f658bd0ce870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:43.269456 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:43.263654 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 31 01:13:43.269456 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:13:43.268251 4440 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.071s" Oct 31 01:13:43.269456 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:43.268347 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tvltr" Oct 31 01:13:43.269456 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:43.268501 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-tvltr" Oct 31 01:13:43.303491 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:43.282524 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" podUID="0b82b50e-bfa7-45b9-8484-c7a331b5ac03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:43.625452 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:43.622524 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" podUID="617e420d-d029-4f4b-8ae5-63fe7d85971e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:43.625452 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:43.623172 4440 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": context deadline exceeded" Oct 31 01:13:43.625452 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:43.623515 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:43.682074 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:43.682027 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:43.813447 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:43.804300 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:43.944872 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:43.944819 4440 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" podUID="617e420d-d029-4f4b-8ae5-63fe7d85971e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": context deadline exceeded" Oct 31 01:13:44.095452 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:44.083377 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:44.096356 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:44.096272 4440 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": context deadline exceeded" Oct 31 01:13:44.100458 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:44.099796 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:44.204656 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:44.198810 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" podUID="2f72d419-0877-4e49-9a85-095ddea8c790" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:44.204656 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:44.199254 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 31 01:13:44.246444 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:44.230899 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:44.303460 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:44.289583 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" podUID="0b82b50e-bfa7-45b9-8484-c7a331b5ac03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:44.303460 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:44.290823 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:44.303460 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:44.300102 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" podUID="cf86081d-7942-454a-8763-f658bd0ce870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 31 01:13:44.637507 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:44.633530 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:44.637507 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:44.633994 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" podUID="617e420d-d029-4f4b-8ae5-63fe7d85971e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": context deadline exceeded" Oct 31 01:13:44.700324 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:44.694527 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:44.815609 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:44.815520 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:45.101584 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:45.101518 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 31 01:13:45.102686 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:45.102643 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:45.205622 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:45.205562 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:45.206361 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:45.205901 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" podUID="2f72d419-0877-4e49-9a85-095ddea8c790" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:45.231591 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:45.231533 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:45.290112 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:45.290055 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" podUID="0b82b50e-bfa7-45b9-8484-c7a331b5ac03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 31 01:13:45.296579 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:45.296528 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:45.301558 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:45.301513 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" podUID="cf86081d-7942-454a-8763-f658bd0ce870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:45.545545 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:45.543655 4440 patch_prober.go:29] interesting pod/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:13:45.545545 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:45.543728 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" podUID="1c523fbe-6cf8-456c-acc0-463ff721c1da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:45.637441 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:45.634186 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:45.637441 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:45.634890 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" podUID="617e420d-d029-4f4b-8ae5-63fe7d85971e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": context deadline exceeded" Oct 31 01:13:45.711445 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:45.698246 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:45.731684 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:45.731639 4440 patch_prober.go:29] interesting pod/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:13:45.732052 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:45.732015 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" podUID="0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:45.828509 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:45.820011 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:45.907237 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:45.906782 4440 patch_prober.go:29] interesting pod/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:13:45.907237 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:45.906861 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" podUID="f8555032-1556-4f20-a9bb-7f1dcbf99dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:46.102665 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:46.102542 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:46.103588 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:46.103550 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:46.130610 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:46.130557 4440 patch_prober.go:29] interesting pod/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:13:46.130904 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:46.130868 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" podUID="961cf83c-7198-4f01-9a04-00d48bb61406" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:46.206574 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:46.206518 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 31 01:13:46.207801 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:46.207768 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" podUID="2f72d419-0877-4e49-9a85-095ddea8c790" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:46.228836 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:46.214489 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:13:46.229523 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:46.229488 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" podUID="dc34b9f5-85b1-4cab-a49e-c2106d685cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:46.229752 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:46.228072 4440 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:46.232033 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:46.232003 4440 patch_prober.go:29] interesting pod/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:13:46.232242 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:46.232206 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" podUID="8f8c2432-9405-4f4f-b24f-1b73ea0da86a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:46.232480 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:46.232447 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:46.304579 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:46.304523 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" podUID="0b82b50e-bfa7-45b9-8484-c7a331b5ac03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:46.305259 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:46.304835 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 31 01:13:46.306332 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:46.304876 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" podUID="cf86081d-7942-454a-8763-f658bd0ce870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:46.435444 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:46.431798 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:46.562447 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:46.558840 4440 patch_prober.go:29] interesting pod/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:13:46.562447 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:46.558922 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" podUID="b29f36f2-9389-4f3c-91cf-658c373b901c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:46.574449 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:46.570262 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj"] Oct 31 01:13:46.660445 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:46.646898 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:46.660445 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:46.637731 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" podUID="617e420d-d029-4f4b-8ae5-63fe7d85971e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:46.700502 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:46.700115 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:46.802458 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:46.799697 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:46.831446 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:46.829483 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw"] Oct 31 01:13:46.884936 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:46.884799 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:46.906663 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:46.906625 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk"] Oct 31 01:13:47.116505 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:47.103104 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:47.159577 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:47.144384 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:47.243446 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:47.214239 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:47.280875 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:47.235648 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:47.315991 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:47.235711 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" podUID="2f72d419-0877-4e49-9a85-095ddea8c790" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Oct 31 01:13:47.323457 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:47.308843 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:47.335904 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:47.308882 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" podUID="0b82b50e-bfa7-45b9-8484-c7a331b5ac03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:47.526461 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:47.308910 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" podUID="cf86081d-7942-454a-8763-f658bd0ce870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:47.526461 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:47.310784 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:47.658864 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:47.657852 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 31 01:13:47.658864 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:47.658327 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" podUID="617e420d-d029-4f4b-8ae5-63fe7d85971e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:47.726456 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:47.700600 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:48.033453 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:47.966172 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:48.128446 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:48.125912 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:48.168455 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:48.159780 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:48.180606 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:48.180552 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:48.245462 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:48.187829 4440 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" podUID="0b82b50e-bfa7-45b9-8484-c7a331b5ac03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:48.271749 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:48.271703 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 31 01:13:48.329644 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:48.329533 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:48.354443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:48.330458 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" podUID="2f72d419-0877-4e49-9a85-095ddea8c790" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:48.354815 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:48.354111 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:48.527454 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:48.514222 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" podUID="0b82b50e-bfa7-45b9-8484-c7a331b5ac03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 31 01:13:48.527687 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:48.527634 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" podUID="cf86081d-7942-454a-8763-f658bd0ce870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 31 01:13:48.663626 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:48.663101 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 31 01:13:48.663626 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:48.663505 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" podUID="617e420d-d029-4f4b-8ae5-63fe7d85971e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": context deadline exceeded" Oct 31 01:13:48.785505 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:48.753519 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:48.961578 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:48.950005 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" Oct 31 01:13:48.996478 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:48.980764 4440 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" podUID="2f72d419-0877-4e49-9a85-095ddea8c790" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:49.103475 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:49.075782 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:49.135430 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:49.135311 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 31 01:13:49.156470 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:49.155670 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" Oct 31 01:13:49.193222 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:49.193178 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" Oct 31 01:13:49.198893 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:49.198869 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" Oct 31 01:13:49.199267 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:49.199251 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" Oct 31 01:13:49.234354 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:49.234259 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" Oct 31 01:13:49.247723 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:49.247691 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" Oct 31 01:13:49.271451 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:49.271390 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" Oct 31 01:13:49.271776 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:49.271761 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" Oct 31 01:13:49.271970 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:49.271961 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" Oct 31 01:13:49.274498 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:49.252739 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 01:13:49.287447 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:49.278913 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" Oct 31 01:13:49.294627 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:49.294461 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" Oct 31 01:13:49.578288 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:49.578198 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:50.345336 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:50.345300 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:50.462037 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:50.461996 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:50.493568 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:50.492083 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10c0b05c-d7ntq"] Oct 31 01:13:50.499577 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:50.498718 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10c0b05c-d7ntq"] Oct 31 01:13:50.499577 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:50.498842 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10c0b05c-d7ntq" Oct 31 01:13:50.513314 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:50.510848 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:13:50.513860 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:50.513798 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zs6l6\"" Oct 31 01:13:50.594497 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:50.594459 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m87rg\" (UniqueName: \"kubernetes.io/projected/c8634e16-a396-4242-9efb-0c398713cb77-kube-api-access-m87rg\") pod \"maas-api-revoke-sub-10c0b05c-d7ntq\" (UID: \"c8634e16-a396-4242-9efb-0c398713cb77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10c0b05c-d7ntq" Oct 31 01:13:50.594671 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:50.594512 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8634e16-a396-4242-9efb-0c398713cb77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10c0b05c-d7ntq\" (UID: \"c8634e16-a396-4242-9efb-0c398713cb77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10c0b05c-d7ntq" Oct 31 01:13:50.594773 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:50.594757 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8634e16-a396-4242-9efb-0c398713cb77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10c0b05c-d7ntq\" (UID: \"c8634e16-a396-4242-9efb-0c398713cb77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10c0b05c-d7ntq" Oct 31 01:13:50.696037 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:50.695997 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8634e16-a396-4242-9efb-0c398713cb77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10c0b05c-d7ntq\" (UID: \"c8634e16-a396-4242-9efb-0c398713cb77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10c0b05c-d7ntq" Oct 31 01:13:50.696248 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:50.696141 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m87rg\" (UniqueName: \"kubernetes.io/projected/c8634e16-a396-4242-9efb-0c398713cb77-kube-api-access-m87rg\") pod \"maas-api-revoke-sub-10c0b05c-d7ntq\" (UID: \"c8634e16-a396-4242-9efb-0c398713cb77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10c0b05c-d7ntq" Oct 31 01:13:50.696248 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:50.696188 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8634e16-a396-4242-9efb-0c398713cb77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10c0b05c-d7ntq\" (UID: \"c8634e16-a396-4242-9efb-0c398713cb77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10c0b05c-d7ntq" Oct 31 01:13:50.697004 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:50.696972 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8634e16-a396-4242-9efb-0c398713cb77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10c0b05c-d7ntq\" (UID: \"c8634e16-a396-4242-9efb-0c398713cb77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10c0b05c-d7ntq" Oct 31 01:13:50.698732 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:50.698705 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8634e16-a396-4242-9efb-0c398713cb77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10c0b05c-d7ntq\" (UID: \"c8634e16-a396-4242-9efb-0c398713cb77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10c0b05c-d7ntq" Oct 31 01:13:50.704700 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:50.704672 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m87rg\" (UniqueName: \"kubernetes.io/projected/c8634e16-a396-4242-9efb-0c398713cb77-kube-api-access-m87rg\") pod \"maas-api-revoke-sub-10c0b05c-d7ntq\" (UID: \"c8634e16-a396-4242-9efb-0c398713cb77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10c0b05c-d7ntq" Oct 31 01:13:50.836250 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:50.836209 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10c0b05c-d7ntq" Oct 31 01:13:51.124488 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:13:51.124378 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8634e16_a396_4242_9efb_0c398713cb77.slice/crio-596bc91f007fea07029064ad331d80ad88e3b4dae03b57cf9b5291bf73f5f677 WatchSource:0}: Error finding container 596bc91f007fea07029064ad331d80ad88e3b4dae03b57cf9b5291bf73f5f677: Status 404 returned error can't find the container with id 596bc91f007fea07029064ad331d80ad88e3b4dae03b57cf9b5291bf73f5f677 Oct 31 01:13:51.125704 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:51.125673 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10c0b05c-d7ntq"] Oct 31 01:13:51.126863 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:51.126843 4440 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 01:13:51.198644 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:51.198607 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:51.362316 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:51.362281 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10c0b05c-d7ntq" event={"ID":"c8634e16-a396-4242-9efb-0c398713cb77","Type":"ContainerStarted","Data":"596bc91f007fea07029064ad331d80ad88e3b4dae03b57cf9b5291bf73f5f677"} Oct 31 01:13:51.936920 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:51.936599 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:52.677011 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:52.676974 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:52.827654 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:52.827589 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-1b8d05-6f65d64999-htfv4"] Oct 31 01:13:52.835342 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:52.835290 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-1b8d05-5c48d56c8c-swf9d"] Oct 31 01:13:52.949839 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:52.949681 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:53.067984 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:53.067945 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:53.864926 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:53.864885 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:54.711273 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:54.711230 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:55.375629 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:55.375533 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:55.434704 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:55.434667 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10c0b05c-d7ntq" event={"ID":"c8634e16-a396-4242-9efb-0c398713cb77","Type":"ContainerStarted","Data":"005b253fdcf97c9b9d34ee752cdd302bf41717ed5dcc3cd5a3130f9253da1ada"} Oct 31 01:13:55.460918 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:55.460859 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10c0b05c-d7ntq" podStartSLOduration=1.855293614 podStartE2EDuration="5.460843027s" totalImagesPullingTime="3.605549413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:55.449183082 +0000 UTC m=+1079.807905400" watchObservedRunningTime="2026-10-31 01:13:55.460843027 +0000 UTC m=+1079.819565342" Oct 31 01:13:56.095169 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:56.095079 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:56.448899 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:56.448861 4440 generic.go:356] "Generic (PLEG): container finished" podID="c8634e16-a396-4242-9efb-0c398713cb77" containerID="005b253fdcf97c9b9d34ee752cdd302bf41717ed5dcc3cd5a3130f9253da1ada" exitCode=0 Oct 31 01:13:56.449312 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:56.448943 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10c0b05c-d7ntq" event={"ID":"c8634e16-a396-4242-9efb-0c398713cb77","Type":"ContainerDied","Data":"005b253fdcf97c9b9d34ee752cdd302bf41717ed5dcc3cd5a3130f9253da1ada"} Oct 31 01:13:56.793208 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:56.793109 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:57.472307 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:57.472258 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:57.665924 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:57.665898 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10c0b05c-d7ntq" Oct 31 01:13:57.777027 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:57.776951 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8634e16-a396-4242-9efb-0c398713cb77-kube-api-access-m87rg\" (UniqueName: \"kubernetes.io/projected/c8634e16-a396-4242-9efb-0c398713cb77-kube-api-access-m87rg\") pod \"c8634e16-a396-4242-9efb-0c398713cb77\" (UID: \"c8634e16-a396-4242-9efb-0c398713cb77\") " Oct 31 01:13:57.777203 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:57.777182 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8634e16-a396-4242-9efb-0c398713cb77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8634e16-a396-4242-9efb-0c398713cb77-maas-api-openshift-ca\") pod \"c8634e16-a396-4242-9efb-0c398713cb77\" (UID: \"c8634e16-a396-4242-9efb-0c398713cb77\") " Oct 31 01:13:57.777413 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:57.777383 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8634e16-a396-4242-9efb-0c398713cb77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8634e16-a396-4242-9efb-0c398713cb77-maas-api-xks-ca\") pod \"c8634e16-a396-4242-9efb-0c398713cb77\" (UID: \"c8634e16-a396-4242-9efb-0c398713cb77\") " Oct 31 01:13:57.777758 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:57.777558 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8634e16-a396-4242-9efb-0c398713cb77-maas-api-openshift-ca" pod "c8634e16-a396-4242-9efb-0c398713cb77" (UID: "c8634e16-a396-4242-9efb-0c398713cb77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:13:57.778682 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:57.778655 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8634e16-a396-4242-9efb-0c398713cb77-maas-api-openshift-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:13:57.779667 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:57.779640 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8634e16-a396-4242-9efb-0c398713cb77-maas-api-xks-ca" pod "c8634e16-a396-4242-9efb-0c398713cb77" (UID: "c8634e16-a396-4242-9efb-0c398713cb77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:13:57.779751 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:57.779731 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8634e16-a396-4242-9efb-0c398713cb77-kube-api-access-m87rg" pod "c8634e16-a396-4242-9efb-0c398713cb77" (UID: "c8634e16-a396-4242-9efb-0c398713cb77"). InnerVolumeSpecName "kube-api-access-m87rg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:13:57.828733 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:57.828679 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-1b8d05-6f65d64999-htfv4" podUID="c80bb0f7-7c51-4344-91d9-a37dc7d21a83" containerName="payload-processing" containerID="cri-o://09c52943d34c819ac6cddf84b089da9d4bbe5ae0929136054a66be43710e2ef7" gracePeriod=25 Oct 31 01:13:57.836480 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:57.836430 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1b8d05-5c48d56c8c-swf9d" podUID="7c3f1628-66ce-4761-80ea-e7654895d913" containerName="payload-pre-processing" containerID="cri-o://714f6ab78d14b43c7b0400742976aa34100b87b05b122814af1fb1b75090f16d" gracePeriod=25 Oct 31 01:13:57.838373 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:57.838344 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:57.886880 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:57.886841 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8634e16-a396-4242-9efb-0c398713cb77-maas-api-xks-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:13:57.886880 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:57.886882 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m87rg\" (UniqueName: \"kubernetes.io/projected/c8634e16-a396-4242-9efb-0c398713cb77-kube-api-access-m87rg\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:13:58.469760 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:58.469728 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10c0b05c-d7ntq" Oct 31 01:13:58.469937 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:58.469754 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10c0b05c-d7ntq" event={"ID":"c8634e16-a396-4242-9efb-0c398713cb77","Type":"ContainerDied","Data":"596bc91f007fea07029064ad331d80ad88e3b4dae03b57cf9b5291bf73f5f677"} Oct 31 01:13:58.469937 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:58.469812 4440 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="596bc91f007fea07029064ad331d80ad88e3b4dae03b57cf9b5291bf73f5f677" Oct 31 01:13:59.172763 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:59.172716 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:59.780703 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:13:59.780664 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:00.386298 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:00.386246 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:01.011045 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:14:01.010996 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:14:01.076821 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:01.076771 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:01.693190 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:01.693150 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:02.413335 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:02.413299 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4"] Oct 31 01:14:02.425507 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:02.425467 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv"] Oct 31 01:14:02.444518 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:02.443220 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq"] Oct 31 01:14:02.509775 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:02.509366 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:02.612110 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:02.612062 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:03.298329 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:03.298281 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:03.345512 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:03.345274 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1b8d05-5c48d56c8c-swf9d" podUID="7c3f1628-66ce-4761-80ea-e7654895d913" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:14:03.346256 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:03.346214 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-1b8d05-6f65d64999-htfv4" podUID="c80bb0f7-7c51-4344-91d9-a37dc7d21a83" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:14:03.831122 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:03.831065 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:04.060064 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.059985 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" containerID="cri-o://ae7ac08bab1f3b15d991bdbc0242d420e6d7e781cd4833f6dad4f2723c4e13dd" gracePeriod=43 Oct 31 01:14:04.110762 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.110713 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" containerID="cri-o://a490227f3a04ca24e002491fed82910c318ffc6d73a7bb60a390977ae0159cb2" gracePeriod=43 Oct 31 01:14:04.158287 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.158253 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:04.249375 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.248809 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": dial tcp 10.133.0.69:8000: connect: connection refused" Oct 31 01:14:04.250611 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.249968 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" containerID="cri-o://91cddc094a32c92a52ca7aea754d54b418916c6acf4e8e501e8f7cf99c150ca9" gracePeriod=43 Oct 31 01:14:04.383493 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:14:04.383166 4440 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc29662e2_8234_4786_9b42_30ca828deb74.slice/crio-conmon-91cddc094a32c92a52ca7aea754d54b418916c6acf4e8e501e8f7cf99c150ca9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc29662e2_8234_4786_9b42_30ca828deb74.slice/crio-91cddc094a32c92a52ca7aea754d54b418916c6acf4e8e501e8f7cf99c150ca9.scope\": RecentStats: unable to find data in memory cache]" Oct 31 01:14:04.537490 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.537452 4440 generic.go:356] "Generic (PLEG): container finished" podID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerID="a490227f3a04ca24e002491fed82910c318ffc6d73a7bb60a390977ae0159cb2" exitCode=0 Oct 31 01:14:04.537791 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.537763 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" event={"ID":"08a5f6da-8cc4-4a5a-9666-fe35c9574f95","Type":"ContainerDied","Data":"a490227f3a04ca24e002491fed82910c318ffc6d73a7bb60a390977ae0159cb2"} Oct 31 01:14:04.537960 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.537942 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" event={"ID":"08a5f6da-8cc4-4a5a-9666-fe35c9574f95","Type":"ContainerDied","Data":"c1b4a1241c4a7c0c09dca06538bc5cee1e3e00d8f0e2aca3b8ec3ed4a02ca520"} Oct 31 01:14:04.538077 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.538062 4440 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1b4a1241c4a7c0c09dca06538bc5cee1e3e00d8f0e2aca3b8ec3ed4a02ca520" Oct 31 01:14:04.541533 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.541507 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" Oct 31 01:14:04.542903 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.542873 4440 generic.go:356] "Generic (PLEG): container finished" podID="c29662e2-8234-4786-9b42-30ca828deb74" containerID="91cddc094a32c92a52ca7aea754d54b418916c6acf4e8e501e8f7cf99c150ca9" exitCode=0 Oct 31 01:14:04.543271 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.543249 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" event={"ID":"c29662e2-8234-4786-9b42-30ca828deb74","Type":"ContainerDied","Data":"91cddc094a32c92a52ca7aea754d54b418916c6acf4e8e501e8f7cf99c150ca9"} Oct 31 01:14:04.545253 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.545232 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" Oct 31 01:14:04.546162 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.546135 4440 generic.go:356] "Generic (PLEG): container finished" podID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerID="ae7ac08bab1f3b15d991bdbc0242d420e6d7e781cd4833f6dad4f2723c4e13dd" exitCode=0 Oct 31 01:14:04.546343 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.546322 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" event={"ID":"1a953e39-91e4-42cc-b7b4-00abdda9e139","Type":"ContainerDied","Data":"ae7ac08bab1f3b15d991bdbc0242d420e6d7e781cd4833f6dad4f2723c4e13dd"} Oct 31 01:14:04.546563 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.546542 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" event={"ID":"1a953e39-91e4-42cc-b7b4-00abdda9e139","Type":"ContainerDied","Data":"0e3bbccb5d8ff85569842ee9469a1164c8224f75039e6cc7f829569918bee5c1"} Oct 31 01:14:04.546693 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.546678 4440 scope.go:122] "RemoveContainer" containerID="ae7ac08bab1f3b15d991bdbc0242d420e6d7e781cd4833f6dad4f2723c4e13dd" Oct 31 01:14:04.546976 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.546956 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw" Oct 31 01:14:04.578402 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.578377 4440 scope.go:122] "RemoveContainer" containerID="ae7ac08bab1f3b15d991bdbc0242d420e6d7e781cd4833f6dad4f2723c4e13dd" Oct 31 01:14:04.578895 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:14:04.578864 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae7ac08bab1f3b15d991bdbc0242d420e6d7e781cd4833f6dad4f2723c4e13dd\": container with ID starting with ae7ac08bab1f3b15d991bdbc0242d420e6d7e781cd4833f6dad4f2723c4e13dd not found: ID does not exist" containerID="ae7ac08bab1f3b15d991bdbc0242d420e6d7e781cd4833f6dad4f2723c4e13dd" Oct 31 01:14:04.579034 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.578981 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae7ac08bab1f3b15d991bdbc0242d420e6d7e781cd4833f6dad4f2723c4e13dd"} err="failed to get container status \"ae7ac08bab1f3b15d991bdbc0242d420e6d7e781cd4833f6dad4f2723c4e13dd\": rpc error: code = NotFound desc = could not find container \"ae7ac08bab1f3b15d991bdbc0242d420e6d7e781cd4833f6dad4f2723c4e13dd\": container with ID starting with ae7ac08bab1f3b15d991bdbc0242d420e6d7e781cd4833f6dad4f2723c4e13dd not found: ID does not exist" Oct 31 01:14:04.628784 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.628684 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a953e39-91e4-42cc-b7b4-00abdda9e139-home\" (UniqueName: \"kubernetes.io/empty-dir/1a953e39-91e4-42cc-b7b4-00abdda9e139-home\") pod \"1a953e39-91e4-42cc-b7b4-00abdda9e139\" (UID: \"1a953e39-91e4-42cc-b7b4-00abdda9e139\") " Oct 31 01:14:04.628978 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.628799 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08a5f6da-8cc4-4a5a-9666-fe35c9574f95-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08a5f6da-8cc4-4a5a-9666-fe35c9574f95-model-cache\") pod \"08a5f6da-8cc4-4a5a-9666-fe35c9574f95\" (UID: \"08a5f6da-8cc4-4a5a-9666-fe35c9574f95\") " Oct 31 01:14:04.629112 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.629088 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a953e39-91e4-42cc-b7b4-00abdda9e139-home" pod "1a953e39-91e4-42cc-b7b4-00abdda9e139" (UID: "1a953e39-91e4-42cc-b7b4-00abdda9e139"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:04.629386 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.629364 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a5f6da-8cc4-4a5a-9666-fe35c9574f95-model-cache" pod "08a5f6da-8cc4-4a5a-9666-fe35c9574f95" (UID: "08a5f6da-8cc4-4a5a-9666-fe35c9574f95"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:04.630198 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.630171 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a953e39-91e4-42cc-b7b4-00abdda9e139-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a953e39-91e4-42cc-b7b4-00abdda9e139-tmp-dir\") pod \"1a953e39-91e4-42cc-b7b4-00abdda9e139\" (UID: \"1a953e39-91e4-42cc-b7b4-00abdda9e139\") " Oct 31 01:14:04.630470 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.630398 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08a5f6da-8cc4-4a5a-9666-fe35c9574f95-home\" (UniqueName: \"kubernetes.io/empty-dir/08a5f6da-8cc4-4a5a-9666-fe35c9574f95-home\") pod \"08a5f6da-8cc4-4a5a-9666-fe35c9574f95\" (UID: \"08a5f6da-8cc4-4a5a-9666-fe35c9574f95\") " Oct 31 01:14:04.630660 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.630640 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a953e39-91e4-42cc-b7b4-00abdda9e139-tmp-dir" pod "1a953e39-91e4-42cc-b7b4-00abdda9e139" (UID: "1a953e39-91e4-42cc-b7b4-00abdda9e139"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:04.632069 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.632032 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a5f6da-8cc4-4a5a-9666-fe35c9574f95-home" pod "08a5f6da-8cc4-4a5a-9666-fe35c9574f95" (UID: "08a5f6da-8cc4-4a5a-9666-fe35c9574f95"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:04.632887 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.632861 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a953e39-91e4-42cc-b7b4-00abdda9e139-kube-api-access-v6ngf\" (UniqueName: \"kubernetes.io/projected/1a953e39-91e4-42cc-b7b4-00abdda9e139-kube-api-access-v6ngf\") pod \"1a953e39-91e4-42cc-b7b4-00abdda9e139\" (UID: \"1a953e39-91e4-42cc-b7b4-00abdda9e139\") " Oct 31 01:14:04.632994 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.632955 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08a5f6da-8cc4-4a5a-9666-fe35c9574f95-tls-certs\" (UniqueName: \"kubernetes.io/secret/08a5f6da-8cc4-4a5a-9666-fe35c9574f95-tls-certs\") pod \"08a5f6da-8cc4-4a5a-9666-fe35c9574f95\" (UID: \"08a5f6da-8cc4-4a5a-9666-fe35c9574f95\") " Oct 31 01:14:04.633052 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.633001 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a953e39-91e4-42cc-b7b4-00abdda9e139-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a953e39-91e4-42cc-b7b4-00abdda9e139-model-cache\") pod \"1a953e39-91e4-42cc-b7b4-00abdda9e139\" (UID: \"1a953e39-91e4-42cc-b7b4-00abdda9e139\") " Oct 31 01:14:04.633111 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.633048 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08a5f6da-8cc4-4a5a-9666-fe35c9574f95-dshm\" (UniqueName: \"kubernetes.io/empty-dir/08a5f6da-8cc4-4a5a-9666-fe35c9574f95-dshm\") pod \"08a5f6da-8cc4-4a5a-9666-fe35c9574f95\" (UID: \"08a5f6da-8cc4-4a5a-9666-fe35c9574f95\") " Oct 31 01:14:04.635186 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.635134 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a953e39-91e4-42cc-b7b4-00abdda9e139-model-cache" pod "1a953e39-91e4-42cc-b7b4-00abdda9e139" (UID: "1a953e39-91e4-42cc-b7b4-00abdda9e139"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:04.636925 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.636902 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08a5f6da-8cc4-4a5a-9666-fe35c9574f95-kube-api-access-n56wx\" (UniqueName: \"kubernetes.io/projected/08a5f6da-8cc4-4a5a-9666-fe35c9574f95-kube-api-access-n56wx\") pod \"08a5f6da-8cc4-4a5a-9666-fe35c9574f95\" (UID: \"08a5f6da-8cc4-4a5a-9666-fe35c9574f95\") " Oct 31 01:14:04.637137 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.637117 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a953e39-91e4-42cc-b7b4-00abdda9e139-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a953e39-91e4-42cc-b7b4-00abdda9e139-dshm\") pod \"1a953e39-91e4-42cc-b7b4-00abdda9e139\" (UID: \"1a953e39-91e4-42cc-b7b4-00abdda9e139\") " Oct 31 01:14:04.637383 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.637368 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08a5f6da-8cc4-4a5a-9666-fe35c9574f95-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08a5f6da-8cc4-4a5a-9666-fe35c9574f95-tmp-dir\") pod \"08a5f6da-8cc4-4a5a-9666-fe35c9574f95\" (UID: \"08a5f6da-8cc4-4a5a-9666-fe35c9574f95\") " Oct 31 01:14:04.639075 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.637661 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a953e39-91e4-42cc-b7b4-00abdda9e139-tls-certs\" (UniqueName: \"kubernetes.io/secret/1a953e39-91e4-42cc-b7b4-00abdda9e139-tls-certs\") pod \"1a953e39-91e4-42cc-b7b4-00abdda9e139\" (UID: \"1a953e39-91e4-42cc-b7b4-00abdda9e139\") " Oct 31 01:14:04.640107 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.638116 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a5f6da-8cc4-4a5a-9666-fe35c9574f95-dshm" pod "08a5f6da-8cc4-4a5a-9666-fe35c9574f95" (UID: "08a5f6da-8cc4-4a5a-9666-fe35c9574f95"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:04.640107 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.638605 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08a5f6da-8cc4-4a5a-9666-fe35c9574f95-tls-certs" pod "08a5f6da-8cc4-4a5a-9666-fe35c9574f95" (UID: "08a5f6da-8cc4-4a5a-9666-fe35c9574f95"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:14:04.640107 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.639006 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a5f6da-8cc4-4a5a-9666-fe35c9574f95-tmp-dir" pod "08a5f6da-8cc4-4a5a-9666-fe35c9574f95" (UID: "08a5f6da-8cc4-4a5a-9666-fe35c9574f95"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:04.644200 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.644173 4440 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a953e39-91e4-42cc-b7b4-00abdda9e139-tmp-dir\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:04.644371 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.644358 4440 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08a5f6da-8cc4-4a5a-9666-fe35c9574f95-home\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:04.644523 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.644511 4440 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08a5f6da-8cc4-4a5a-9666-fe35c9574f95-tls-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:04.644630 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.644614 4440 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a953e39-91e4-42cc-b7b4-00abdda9e139-model-cache\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:04.644730 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.644705 4440 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08a5f6da-8cc4-4a5a-9666-fe35c9574f95-dshm\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:04.644809 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.644799 4440 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08a5f6da-8cc4-4a5a-9666-fe35c9574f95-tmp-dir\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:04.644930 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.644920 4440 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a953e39-91e4-42cc-b7b4-00abdda9e139-home\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:04.645008 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.644998 4440 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08a5f6da-8cc4-4a5a-9666-fe35c9574f95-model-cache\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:04.650945 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.650795 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a953e39-91e4-42cc-b7b4-00abdda9e139-kube-api-access-v6ngf" pod "1a953e39-91e4-42cc-b7b4-00abdda9e139" (UID: "1a953e39-91e4-42cc-b7b4-00abdda9e139"). InnerVolumeSpecName "kube-api-access-v6ngf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:04.650945 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.650837 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a953e39-91e4-42cc-b7b4-00abdda9e139-dshm" pod "1a953e39-91e4-42cc-b7b4-00abdda9e139" (UID: "1a953e39-91e4-42cc-b7b4-00abdda9e139"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:04.650945 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.650879 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08a5f6da-8cc4-4a5a-9666-fe35c9574f95-kube-api-access-n56wx" pod "08a5f6da-8cc4-4a5a-9666-fe35c9574f95" (UID: "08a5f6da-8cc4-4a5a-9666-fe35c9574f95"). InnerVolumeSpecName "kube-api-access-n56wx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:04.675076 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.675034 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a953e39-91e4-42cc-b7b4-00abdda9e139-tls-certs" pod "1a953e39-91e4-42cc-b7b4-00abdda9e139" (UID: "1a953e39-91e4-42cc-b7b4-00abdda9e139"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:14:04.692568 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.692527 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" Oct 31 01:14:04.746763 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.746713 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6ngf\" (UniqueName: \"kubernetes.io/projected/1a953e39-91e4-42cc-b7b4-00abdda9e139-kube-api-access-v6ngf\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:04.746763 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.746766 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n56wx\" (UniqueName: \"kubernetes.io/projected/08a5f6da-8cc4-4a5a-9666-fe35c9574f95-kube-api-access-n56wx\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:04.746980 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.746783 4440 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a953e39-91e4-42cc-b7b4-00abdda9e139-dshm\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:04.746980 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.746798 4440 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a953e39-91e4-42cc-b7b4-00abdda9e139-tls-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:04.855557 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.855517 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c29662e2-8234-4786-9b42-30ca828deb74-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c29662e2-8234-4786-9b42-30ca828deb74-dshm\") pod \"c29662e2-8234-4786-9b42-30ca828deb74\" (UID: \"c29662e2-8234-4786-9b42-30ca828deb74\") " Oct 31 01:14:04.855814 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.855682 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c29662e2-8234-4786-9b42-30ca828deb74-home\" (UniqueName: \"kubernetes.io/empty-dir/c29662e2-8234-4786-9b42-30ca828deb74-home\") pod \"c29662e2-8234-4786-9b42-30ca828deb74\" (UID: \"c29662e2-8234-4786-9b42-30ca828deb74\") " Oct 31 01:14:04.855814 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.855773 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c29662e2-8234-4786-9b42-30ca828deb74-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c29662e2-8234-4786-9b42-30ca828deb74-model-cache\") pod \"c29662e2-8234-4786-9b42-30ca828deb74\" (UID: \"c29662e2-8234-4786-9b42-30ca828deb74\") " Oct 31 01:14:04.855814 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.855806 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c29662e2-8234-4786-9b42-30ca828deb74-tls-certs\" (UniqueName: \"kubernetes.io/secret/c29662e2-8234-4786-9b42-30ca828deb74-tls-certs\") pod \"c29662e2-8234-4786-9b42-30ca828deb74\" (UID: \"c29662e2-8234-4786-9b42-30ca828deb74\") " Oct 31 01:14:04.856414 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.855912 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c29662e2-8234-4786-9b42-30ca828deb74-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c29662e2-8234-4786-9b42-30ca828deb74-tmp-dir\") pod \"c29662e2-8234-4786-9b42-30ca828deb74\" (UID: \"c29662e2-8234-4786-9b42-30ca828deb74\") " Oct 31 01:14:04.856414 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.856086 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c29662e2-8234-4786-9b42-30ca828deb74-kube-api-access-6vbr4\" (UniqueName: \"kubernetes.io/projected/c29662e2-8234-4786-9b42-30ca828deb74-kube-api-access-6vbr4\") pod \"c29662e2-8234-4786-9b42-30ca828deb74\" (UID: \"c29662e2-8234-4786-9b42-30ca828deb74\") " Oct 31 01:14:04.858919 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.858870 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29662e2-8234-4786-9b42-30ca828deb74-home" pod "c29662e2-8234-4786-9b42-30ca828deb74" (UID: "c29662e2-8234-4786-9b42-30ca828deb74"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:04.859059 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.858943 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29662e2-8234-4786-9b42-30ca828deb74-model-cache" pod "c29662e2-8234-4786-9b42-30ca828deb74" (UID: "c29662e2-8234-4786-9b42-30ca828deb74"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:04.859196 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.859174 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29662e2-8234-4786-9b42-30ca828deb74-tmp-dir" pod "c29662e2-8234-4786-9b42-30ca828deb74" (UID: "c29662e2-8234-4786-9b42-30ca828deb74"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:04.861054 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.861019 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c29662e2-8234-4786-9b42-30ca828deb74-kube-api-access-6vbr4" pod "c29662e2-8234-4786-9b42-30ca828deb74" (UID: "c29662e2-8234-4786-9b42-30ca828deb74"). InnerVolumeSpecName "kube-api-access-6vbr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:04.862978 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.862950 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c29662e2-8234-4786-9b42-30ca828deb74-tls-certs" pod "c29662e2-8234-4786-9b42-30ca828deb74" (UID: "c29662e2-8234-4786-9b42-30ca828deb74"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:14:04.863179 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.863147 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29662e2-8234-4786-9b42-30ca828deb74-dshm" pod "c29662e2-8234-4786-9b42-30ca828deb74" (UID: "c29662e2-8234-4786-9b42-30ca828deb74"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:04.879549 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.879463 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw"] Oct 31 01:14:04.884657 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.884625 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-1b8d05/distinct-w6-1b8d05-kserve-77976c577f-qn6zw"] Oct 31 01:14:04.958361 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.958323 4440 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c29662e2-8234-4786-9b42-30ca828deb74-tmp-dir\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:04.958361 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.958359 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vbr4\" (UniqueName: \"kubernetes.io/projected/c29662e2-8234-4786-9b42-30ca828deb74-kube-api-access-6vbr4\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:04.958617 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.958373 4440 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c29662e2-8234-4786-9b42-30ca828deb74-dshm\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:04.958617 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.958382 4440 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c29662e2-8234-4786-9b42-30ca828deb74-home\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:04.958617 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.958389 4440 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c29662e2-8234-4786-9b42-30ca828deb74-model-cache\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:04.958617 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:04.958398 4440 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c29662e2-8234-4786-9b42-30ca828deb74-tls-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:05.167213 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:05.167118 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:05.556574 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:05.556541 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" Oct 31 01:14:05.557050 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:05.556539 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj" event={"ID":"c29662e2-8234-4786-9b42-30ca828deb74","Type":"ContainerDied","Data":"b2d9e159ddd92f632c353eb0bb43bf6f5b893237cde91601daa138f0f75e26b3"} Oct 31 01:14:05.557050 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:05.556680 4440 scope.go:122] "RemoveContainer" containerID="91cddc094a32c92a52ca7aea754d54b418916c6acf4e8e501e8f7cf99c150ca9" Oct 31 01:14:05.558557 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:05.558534 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk" Oct 31 01:14:05.586080 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:05.586039 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj"] Oct 31 01:14:05.591030 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:05.590992 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-1b8d05/distinct-2-w6-1b8d05-kserve-cc96ffbb4-8f7wj"] Oct 31 01:14:05.601230 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:05.600362 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk"] Oct 31 01:14:05.605750 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:05.605711 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-1b8d05/premium-w6-1b8d05-kserve-b68468ddd-xv5fk"] Oct 31 01:14:05.773780 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:05.773734 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" path="/var/lib/kubelet/pods/08a5f6da-8cc4-4a5a-9666-fe35c9574f95/volumes" Oct 31 01:14:05.774336 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:05.774315 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" path="/var/lib/kubelet/pods/1a953e39-91e4-42cc-b7b4-00abdda9e139/volumes" Oct 31 01:14:05.774878 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:05.774856 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c29662e2-8234-4786-9b42-30ca828deb74" path="/var/lib/kubelet/pods/c29662e2-8234-4786-9b42-30ca828deb74/volumes" Oct 31 01:14:05.898437 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:05.898320 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:06.594782 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:06.594745 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:07.401049 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:07.401009 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:07.892452 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:07.892274 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-0a6cca-6599d4945f-dlflj"] Oct 31 01:14:07.983100 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:07.976811 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:08.078896 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:08.078850 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:08.364470 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:08.360933 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-1b8d05-5fdbf4bd8c-lf5w4"] Oct 31 01:14:08.364470 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:08.361220 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1b8d05-5fdbf4bd8c-lf5w4" podUID="293b71e1-036c-491c-b693-f7acc63bed0f" containerName="maas-api" containerID="cri-o://63c1d7b949c20ade4dc507c1aed0c655135f40c64f0952f91fddc1c91dabc118" gracePeriod=30 Oct 31 01:14:08.409958 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:08.406692 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:08.788831 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:08.788780 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx"] Oct 31 01:14:08.797600 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:08.797559 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w"] Oct 31 01:14:08.892029 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:08.891875 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:08.953112 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:08.951387 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1b8d05-5fdbf4bd8c-lf5w4" Oct 31 01:14:09.018034 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.017996 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:09.093727 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.093694 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/293b71e1-036c-491c-b693-f7acc63bed0f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/293b71e1-036c-491c-b693-f7acc63bed0f-metrics-certs\") pod \"293b71e1-036c-491c-b693-f7acc63bed0f\" (UID: \"293b71e1-036c-491c-b693-f7acc63bed0f\") " Oct 31 01:14:09.093956 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.093846 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/293b71e1-036c-491c-b693-f7acc63bed0f-kube-api-access-t4pcb\" (UniqueName: \"kubernetes.io/projected/293b71e1-036c-491c-b693-f7acc63bed0f-kube-api-access-t4pcb\") pod \"293b71e1-036c-491c-b693-f7acc63bed0f\" (UID: \"293b71e1-036c-491c-b693-f7acc63bed0f\") " Oct 31 01:14:09.093956 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.093923 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/293b71e1-036c-491c-b693-f7acc63bed0f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/293b71e1-036c-491c-b693-f7acc63bed0f-maas-api-tls\") pod \"293b71e1-036c-491c-b693-f7acc63bed0f\" (UID: \"293b71e1-036c-491c-b693-f7acc63bed0f\") " Oct 31 01:14:09.094081 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.094030 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/293b71e1-036c-491c-b693-f7acc63bed0f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/293b71e1-036c-491c-b693-f7acc63bed0f-openshift-service-ca\") pod \"293b71e1-036c-491c-b693-f7acc63bed0f\" (UID: \"293b71e1-036c-491c-b693-f7acc63bed0f\") " Oct 31 01:14:09.095071 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.095035 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/293b71e1-036c-491c-b693-f7acc63bed0f-openshift-service-ca" pod "293b71e1-036c-491c-b693-f7acc63bed0f" (UID: "293b71e1-036c-491c-b693-f7acc63bed0f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:09.122499 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.097872 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/293b71e1-036c-491c-b693-f7acc63bed0f-maas-api-tls" pod "293b71e1-036c-491c-b693-f7acc63bed0f" (UID: "293b71e1-036c-491c-b693-f7acc63bed0f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:14:09.144412 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.143656 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/293b71e1-036c-491c-b693-f7acc63bed0f-kube-api-access-t4pcb" pod "293b71e1-036c-491c-b693-f7acc63bed0f" (UID: "293b71e1-036c-491c-b693-f7acc63bed0f"). InnerVolumeSpecName "kube-api-access-t4pcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:09.144412 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.143770 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/293b71e1-036c-491c-b693-f7acc63bed0f-metrics-certs" pod "293b71e1-036c-491c-b693-f7acc63bed0f" (UID: "293b71e1-036c-491c-b693-f7acc63bed0f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:14:09.215443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.196837 4440 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/293b71e1-036c-491c-b693-f7acc63bed0f-metrics-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:09.215794 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.215761 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4pcb\" (UniqueName: \"kubernetes.io/projected/293b71e1-036c-491c-b693-f7acc63bed0f-kube-api-access-t4pcb\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:09.215911 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.215897 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/293b71e1-036c-491c-b693-f7acc63bed0f-maas-api-tls\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:09.216040 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.216026 4440 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/293b71e1-036c-491c-b693-f7acc63bed0f-openshift-service-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:09.334618 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.333323 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:09.554440 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.548251 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d7a7c8d-hg9b5"] Oct 31 01:14:09.554440 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.549320 4440 cpu_manager.go:414] "containerMap: removing container" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" Oct 31 01:14:09.554440 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.549902 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" Oct 31 01:14:09.554440 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.549946 4440 cpu_manager.go:414] "containerMap: removing container" podUID="c8634e16-a396-4242-9efb-0c398713cb77" containerName="revoke-keys" Oct 31 01:14:09.554440 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.549954 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8634e16-a396-4242-9efb-0c398713cb77" containerName="revoke-keys" Oct 31 01:14:09.554440 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.549972 4440 cpu_manager.go:414] "containerMap: removing container" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" Oct 31 01:14:09.554440 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.549980 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" Oct 31 01:14:09.554440 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.549992 4440 cpu_manager.go:414] "containerMap: removing container" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" Oct 31 01:14:09.554440 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.549999 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" Oct 31 01:14:09.554440 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.550051 4440 cpu_manager.go:414] "containerMap: removing container" podUID="293b71e1-036c-491c-b693-f7acc63bed0f" containerName="maas-api" Oct 31 01:14:09.554440 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.550057 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="293b71e1-036c-491c-b693-f7acc63bed0f" containerName="maas-api" Oct 31 01:14:09.554440 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.550245 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="08a5f6da-8cc4-4a5a-9666-fe35c9574f95" containerName="main" Oct 31 01:14:09.554440 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.550261 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8634e16-a396-4242-9efb-0c398713cb77" containerName="revoke-keys" Oct 31 01:14:09.554440 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.550271 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="c29662e2-8234-4786-9b42-30ca828deb74" containerName="main" Oct 31 01:14:09.554440 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.550280 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a953e39-91e4-42cc-b7b4-00abdda9e139" containerName="main" Oct 31 01:14:09.554440 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.550294 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="293b71e1-036c-491c-b693-f7acc63bed0f" containerName="maas-api" Oct 31 01:14:09.568670 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.568597 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:09.568866 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.568817 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d7a7c8d-hg9b5" Oct 31 01:14:09.571643 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.571603 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d7a7c8d-hg9b5"] Oct 31 01:14:09.612443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.611342 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:14:09.612443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.611550 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zs6l6\"" Oct 31 01:14:09.665760 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.665718 4440 generic.go:356] "Generic (PLEG): container finished" podID="293b71e1-036c-491c-b693-f7acc63bed0f" containerID="63c1d7b949c20ade4dc507c1aed0c655135f40c64f0952f91fddc1c91dabc118" exitCode=0 Oct 31 01:14:09.665951 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.665775 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1b8d05-5fdbf4bd8c-lf5w4" Oct 31 01:14:09.665951 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.665795 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1b8d05-5fdbf4bd8c-lf5w4" event={"ID":"293b71e1-036c-491c-b693-f7acc63bed0f","Type":"ContainerDied","Data":"63c1d7b949c20ade4dc507c1aed0c655135f40c64f0952f91fddc1c91dabc118"} Oct 31 01:14:09.665951 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.665857 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1b8d05-5fdbf4bd8c-lf5w4" event={"ID":"293b71e1-036c-491c-b693-f7acc63bed0f","Type":"ContainerDied","Data":"0edf3ba2cdcb0226dc4f2036a76ba0b28da23e551af26f1ad8a8f6de410e9536"} Oct 31 01:14:09.665951 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.665889 4440 scope.go:122] "RemoveContainer" containerID="63c1d7b949c20ade4dc507c1aed0c655135f40c64f0952f91fddc1c91dabc118" Oct 31 01:14:09.695238 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.695214 4440 scope.go:122] "RemoveContainer" containerID="63c1d7b949c20ade4dc507c1aed0c655135f40c64f0952f91fddc1c91dabc118" Oct 31 01:14:09.695685 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:14:09.695657 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63c1d7b949c20ade4dc507c1aed0c655135f40c64f0952f91fddc1c91dabc118\": container with ID starting with 63c1d7b949c20ade4dc507c1aed0c655135f40c64f0952f91fddc1c91dabc118 not found: ID does not exist" containerID="63c1d7b949c20ade4dc507c1aed0c655135f40c64f0952f91fddc1c91dabc118" Oct 31 01:14:09.695799 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.695693 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63c1d7b949c20ade4dc507c1aed0c655135f40c64f0952f91fddc1c91dabc118"} err="failed to get container status \"63c1d7b949c20ade4dc507c1aed0c655135f40c64f0952f91fddc1c91dabc118\": rpc error: code = NotFound desc = could not find container \"63c1d7b949c20ade4dc507c1aed0c655135f40c64f0952f91fddc1c91dabc118\": container with ID starting with 63c1d7b949c20ade4dc507c1aed0c655135f40c64f0952f91fddc1c91dabc118 not found: ID does not exist" Oct 31 01:14:09.713802 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.713728 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-1b8d05-5fdbf4bd8c-lf5w4"] Oct 31 01:14:09.719343 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.719304 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-1b8d05-5fdbf4bd8c-lf5w4"] Oct 31 01:14:09.726826 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.726787 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/478a9053-b8a9-4b70-a96d-aa50644aaf53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d7a7c8d-hg9b5\" (UID: \"478a9053-b8a9-4b70-a96d-aa50644aaf53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d7a7c8d-hg9b5" Oct 31 01:14:09.727028 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.726846 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzqj9\" (UniqueName: \"kubernetes.io/projected/478a9053-b8a9-4b70-a96d-aa50644aaf53-kube-api-access-nzqj9\") pod \"maas-api-revoke-sub-0d7a7c8d-hg9b5\" (UID: \"478a9053-b8a9-4b70-a96d-aa50644aaf53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d7a7c8d-hg9b5" Oct 31 01:14:09.727089 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.727066 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/478a9053-b8a9-4b70-a96d-aa50644aaf53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d7a7c8d-hg9b5\" (UID: \"478a9053-b8a9-4b70-a96d-aa50644aaf53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d7a7c8d-hg9b5" Oct 31 01:14:09.774488 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.774432 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="293b71e1-036c-491c-b693-f7acc63bed0f" path="/var/lib/kubelet/pods/293b71e1-036c-491c-b693-f7acc63bed0f/volumes" Oct 31 01:14:09.828783 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.828687 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/478a9053-b8a9-4b70-a96d-aa50644aaf53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d7a7c8d-hg9b5\" (UID: \"478a9053-b8a9-4b70-a96d-aa50644aaf53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d7a7c8d-hg9b5" Oct 31 01:14:09.829019 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.828996 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/478a9053-b8a9-4b70-a96d-aa50644aaf53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d7a7c8d-hg9b5\" (UID: \"478a9053-b8a9-4b70-a96d-aa50644aaf53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d7a7c8d-hg9b5" Oct 31 01:14:09.829074 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.829056 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzqj9\" (UniqueName: \"kubernetes.io/projected/478a9053-b8a9-4b70-a96d-aa50644aaf53-kube-api-access-nzqj9\") pod \"maas-api-revoke-sub-0d7a7c8d-hg9b5\" (UID: \"478a9053-b8a9-4b70-a96d-aa50644aaf53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d7a7c8d-hg9b5" Oct 31 01:14:09.829539 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.829518 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/478a9053-b8a9-4b70-a96d-aa50644aaf53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d7a7c8d-hg9b5\" (UID: \"478a9053-b8a9-4b70-a96d-aa50644aaf53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d7a7c8d-hg9b5" Oct 31 01:14:09.831732 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.831704 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/478a9053-b8a9-4b70-a96d-aa50644aaf53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d7a7c8d-hg9b5\" (UID: \"478a9053-b8a9-4b70-a96d-aa50644aaf53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d7a7c8d-hg9b5" Oct 31 01:14:09.837208 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.837182 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzqj9\" (UniqueName: \"kubernetes.io/projected/478a9053-b8a9-4b70-a96d-aa50644aaf53-kube-api-access-nzqj9\") pod \"maas-api-revoke-sub-0d7a7c8d-hg9b5\" (UID: \"478a9053-b8a9-4b70-a96d-aa50644aaf53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d7a7c8d-hg9b5" Oct 31 01:14:09.923270 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:09.923232 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d7a7c8d-hg9b5" Oct 31 01:14:10.026171 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:10.026132 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:10.210800 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:10.210767 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d7a7c8d-hg9b5"] Oct 31 01:14:10.211022 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:14:10.210980 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod478a9053_b8a9_4b70_a96d_aa50644aaf53.slice/crio-be9b872735100cbb0fa0ad88cf00cba5a64bbd5cc49f2e9205e73ee1058e4f2e WatchSource:0}: Error finding container be9b872735100cbb0fa0ad88cf00cba5a64bbd5cc49f2e9205e73ee1058e4f2e: Status 404 returned error can't find the container with id be9b872735100cbb0fa0ad88cf00cba5a64bbd5cc49f2e9205e73ee1058e4f2e Oct 31 01:14:10.678974 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:10.678941 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d7a7c8d-hg9b5" event={"ID":"478a9053-b8a9-4b70-a96d-aa50644aaf53","Type":"ContainerStarted","Data":"3068471fdc47c64e56c6a2631b1637ee76f1f1cda819149bf2c578dca9223b03"} Oct 31 01:14:10.679157 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:10.678992 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d7a7c8d-hg9b5" event={"ID":"478a9053-b8a9-4b70-a96d-aa50644aaf53","Type":"ContainerStarted","Data":"be9b872735100cbb0fa0ad88cf00cba5a64bbd5cc49f2e9205e73ee1058e4f2e"} Oct 31 01:14:10.699769 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:10.699539 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:10.702274 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:10.702216 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d7a7c8d-hg9b5" podStartSLOduration=1.702197917 podStartE2EDuration="1.702197917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:14:10.702156454 +0000 UTC m=+1095.060878773" watchObservedRunningTime="2026-10-31 01:14:10.702197917 +0000 UTC m=+1095.060920234" Oct 31 01:14:11.489922 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:11.489822 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:11.796507 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:11.795920 4440 generic.go:356] "Generic (PLEG): container finished" podID="478a9053-b8a9-4b70-a96d-aa50644aaf53" containerID="3068471fdc47c64e56c6a2631b1637ee76f1f1cda819149bf2c578dca9223b03" exitCode=0 Oct 31 01:14:11.801891 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:11.797899 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d7a7c8d-hg9b5" event={"ID":"478a9053-b8a9-4b70-a96d-aa50644aaf53","Type":"ContainerDied","Data":"3068471fdc47c64e56c6a2631b1637ee76f1f1cda819149bf2c578dca9223b03"} Oct 31 01:14:13.372179 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:13.372129 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-1b8d05-6f65d64999-htfv4" podUID="c80bb0f7-7c51-4344-91d9-a37dc7d21a83" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:14:13.375038 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:13.374986 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1b8d05-5c48d56c8c-swf9d" podUID="7c3f1628-66ce-4761-80ea-e7654895d913" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:14:13.624690 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:13.624512 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-0a6cca-6599d4945f-dlflj" podUID="eaad3e7d-1293-43d5-9fe6-674e69ea840e" containerName="payload-pre-processing" containerID="cri-o://326d064c31cabc994f3860adf0c6c71f24c1ae6bdcd13f040b76733cd4c943c3" gracePeriod=25 Oct 31 01:14:13.666014 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:13.653548 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" podUID="617e420d-d029-4f4b-8ae5-63fe7d85971e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": context deadline exceeded" Oct 31 01:14:13.726559 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:13.720556 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:13.811445 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:13.761704 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:14.299049 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:14.298574 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" podUID="0b82b50e-bfa7-45b9-8484-c7a331b5ac03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:14.366449 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:14.329619 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" podUID="cf86081d-7942-454a-8763-f658bd0ce870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:14.366449 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:14.340521 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:14.366449 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:14.340994 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 31 01:14:14.366449 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:14.341148 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 01:14:14.558369 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:14.525493 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" podUID="2f72d419-0877-4e49-9a85-095ddea8c790" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Oct 31 01:14:14.569696 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:14.569571 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-0a6cca-6599d4945f-dlflj" podUID="eaad3e7d-1293-43d5-9fe6-674e69ea840e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:14:14.671455 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:14.656660 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" podUID="617e420d-d029-4f4b-8ae5-63fe7d85971e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:14.671455 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:14.659984 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" Oct 31 01:14:14.745454 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:14.740339 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:14.751217 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:14.750017 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" Oct 31 01:14:14.782504 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:14.768643 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:14.782504 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:14.770653 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" Oct 31 01:14:14.827683 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:14.827605 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d7a7c8d-hg9b5" Oct 31 01:14:15.025179 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:14.985590 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:15.225844 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:15.096112 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/478a9053-b8a9-4b70-a96d-aa50644aaf53-kube-api-access-nzqj9\" (UniqueName: \"kubernetes.io/projected/478a9053-b8a9-4b70-a96d-aa50644aaf53-kube-api-access-nzqj9\") pod \"478a9053-b8a9-4b70-a96d-aa50644aaf53\" (UID: \"478a9053-b8a9-4b70-a96d-aa50644aaf53\") " Oct 31 01:14:15.262926 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:15.262892 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/478a9053-b8a9-4b70-a96d-aa50644aaf53-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/478a9053-b8a9-4b70-a96d-aa50644aaf53-maas-api-xks-ca\") pod \"478a9053-b8a9-4b70-a96d-aa50644aaf53\" (UID: \"478a9053-b8a9-4b70-a96d-aa50644aaf53\") " Oct 31 01:14:15.263262 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:15.263238 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/478a9053-b8a9-4b70-a96d-aa50644aaf53-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/478a9053-b8a9-4b70-a96d-aa50644aaf53-maas-api-openshift-ca\") pod \"478a9053-b8a9-4b70-a96d-aa50644aaf53\" (UID: \"478a9053-b8a9-4b70-a96d-aa50644aaf53\") " Oct 31 01:14:15.264927 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:15.264893 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/478a9053-b8a9-4b70-a96d-aa50644aaf53-maas-api-openshift-ca" pod "478a9053-b8a9-4b70-a96d-aa50644aaf53" (UID: "478a9053-b8a9-4b70-a96d-aa50644aaf53"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:15.299348 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:15.299304 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" podUID="0b82b50e-bfa7-45b9-8484-c7a331b5ac03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:15.300612 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:15.300584 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" Oct 31 01:14:15.358521 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:15.354505 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:15.370449 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:15.365496 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:15.370449 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:15.366411 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" podUID="cf86081d-7942-454a-8763-f658bd0ce870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 31 01:14:15.386458 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:15.386398 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 31 01:14:15.387397 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:15.386743 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" Oct 31 01:14:15.387397 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:15.386807 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" Oct 31 01:14:15.387397 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:15.386848 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" Oct 31 01:14:15.387397 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:15.386884 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" Oct 31 01:14:15.442640 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:15.442455 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:15.461470 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:15.457630 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d7a7c8d-hg9b5" event={"ID":"478a9053-b8a9-4b70-a96d-aa50644aaf53","Type":"ContainerDied","Data":"be9b872735100cbb0fa0ad88cf00cba5a64bbd5cc49f2e9205e73ee1058e4f2e"} Oct 31 01:14:15.461470 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:15.457703 4440 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be9b872735100cbb0fa0ad88cf00cba5a64bbd5cc49f2e9205e73ee1058e4f2e" Oct 31 01:14:15.461470 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:15.457788 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d7a7c8d-hg9b5" Oct 31 01:14:15.544290 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:15.540998 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" podUID="2f72d419-0877-4e49-9a85-095ddea8c790" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:15.544290 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:15.541699 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" Oct 31 01:14:15.704470 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:15.700506 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" podUID="617e420d-d029-4f4b-8ae5-63fe7d85971e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": context deadline exceeded" Oct 31 01:14:15.739195 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:15.704982 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/478a9053-b8a9-4b70-a96d-aa50644aaf53-maas-api-openshift-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:15.795500 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:15.789969 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:15.851508 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:15.841564 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 31 01:14:16.137510 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:15.946742 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/478a9053-b8a9-4b70-a96d-aa50644aaf53-kube-api-access-nzqj9" pod "478a9053-b8a9-4b70-a96d-aa50644aaf53" (UID: "478a9053-b8a9-4b70-a96d-aa50644aaf53"). InnerVolumeSpecName "kube-api-access-nzqj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:16.137510 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:16.034694 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzqj9\" (UniqueName: \"kubernetes.io/projected/478a9053-b8a9-4b70-a96d-aa50644aaf53-kube-api-access-nzqj9\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:16.137510 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:16.091512 4440 patch_prober.go:29] interesting pod/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:14:16.137510 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:16.091584 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" podUID="f8555032-1556-4f20-a9bb-7f1dcbf99dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:16.169441 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:16.162566 4440 patch_prober.go:29] interesting pod/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:14:16.169441 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:16.162648 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" podUID="961cf83c-7198-4f01-9a04-00d48bb61406" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:16.295672 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:16.295628 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/478a9053-b8a9-4b70-a96d-aa50644aaf53-maas-api-xks-ca" pod "478a9053-b8a9-4b70-a96d-aa50644aaf53" (UID: "478a9053-b8a9-4b70-a96d-aa50644aaf53"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:14:16.310493 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:16.302656 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" podUID="0b82b50e-bfa7-45b9-8484-c7a331b5ac03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:16.310672 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:16.306766 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:14:16.310672 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:16.310629 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" podUID="dc34b9f5-85b1-4cab-a49e-c2106d685cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:16.357600 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:16.357535 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:16.387783 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:16.387664 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:16.388683 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:16.388656 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" podUID="cf86081d-7942-454a-8763-f658bd0ce870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 31 01:14:16.397038 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:16.396986 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:16.429438 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:16.429345 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/478a9053-b8a9-4b70-a96d-aa50644aaf53-maas-api-xks-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:16.517571 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:16.504440 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8499c8-5c65ddd74c-9r5n8"] Oct 31 01:14:16.878453 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:16.844946 4440 patch_prober.go:29] interesting pod/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:14:16.878453 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:16.845035 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" podUID="81095022-acb2-4b1c-acc6-a9501b0ea478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:16.920599 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:16.920552 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" podUID="617e420d-d029-4f4b-8ae5-63fe7d85971e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": context deadline exceeded" Oct 31 01:14:16.921331 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:16.921293 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" podUID="2f72d419-0877-4e49-9a85-095ddea8c790" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:16.970452 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:16.966933 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:17.159987 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:17.065725 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:17.323410 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:17.323363 4440 patch_prober.go:29] interesting pod/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:14:17.323810 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:17.323690 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" podUID="7ee3eb69-94de-447c-b9ec-194a0458f2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:17.497738 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:17.497684 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" podUID="617e420d-d029-4f4b-8ae5-63fe7d85971e" containerName="main" containerID="cri-o://48f30cd271402c703cf948c7a6175c8b5b2778e52a098ec52cae25c96ee06d4b" gracePeriod=45 Oct 31 01:14:17.499102 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:17.498969 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" containerID="cri-o://7ab14572d848ec37a73de36151f934ccb2da98f4021a7dca143227067ade51be" gracePeriod=45 Oct 31 01:14:17.568652 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:17.523027 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" podUID="0b82b50e-bfa7-45b9-8484-c7a331b5ac03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 31 01:14:17.568652 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:17.544883 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-acdaed-55478d5599-md22s" podUID="e4160358-c655-4a6b-bb39-fc8677804241" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.62:50052\" within 1s: context deadline exceeded" Oct 31 01:14:17.655444 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:17.649859 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" podUID="617e420d-d029-4f4b-8ae5-63fe7d85971e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": read tcp 10.133.0.2:34690->10.133.0.87:8000: read: connection reset by peer" Oct 31 01:14:17.655444 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:17.652621 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" podUID="617e420d-d029-4f4b-8ae5-63fe7d85971e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": dial tcp 10.133.0.87:8000: connect: connection refused" Oct 31 01:14:17.666454 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:17.663925 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" podUID="cf86081d-7942-454a-8763-f658bd0ce870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:17.666454 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:17.664698 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 01:14:17.666454 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:17.665757 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:17.738445 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:17.737045 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" containerID="cri-o://e87b951688d5a10a0c118c8185589d2c6ac8eb73c32557d515181916a0cc1437" gracePeriod=45 Oct 31 01:14:17.820364 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:17.820311 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:18.024673 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:17.932764 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" podUID="2f72d419-0877-4e49-9a85-095ddea8c790" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:18.111739 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:18.111669 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:18.160506 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:18.160460 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 31 01:14:18.306313 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:18.306208 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:18.554474 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:18.547057 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:18.569941 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:18.569828 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" podUID="0b82b50e-bfa7-45b9-8484-c7a331b5ac03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:18.739606 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:18.708894 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:18.739606 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:18.716019 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" podUID="cf86081d-7942-454a-8763-f658bd0ce870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:18.741069 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:18.741023 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 01:14:18.839569 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:14:18.782868 4440 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.019s" Oct 31 01:14:18.953447 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:18.849819 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:18.953447 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:18.891390 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" podUID="617e420d-d029-4f4b-8ae5-63fe7d85971e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": dial tcp 10.133.0.87:8000: connect: connection refused" Oct 31 01:14:18.953447 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:18.942210 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" podUID="2f72d419-0877-4e49-9a85-095ddea8c790" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Oct 31 01:14:19.130197 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:19.130069 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:19.174491 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:19.174071 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:19.572443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:19.559471 4440 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded" Oct 31 01:14:19.595448 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:19.593838 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" podUID="0b82b50e-bfa7-45b9-8484-c7a331b5ac03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:19.636228 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:19.636180 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" podUID="617e420d-d029-4f4b-8ae5-63fe7d85971e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": dial tcp 10.133.0.87:8000: connect: connection refused" Oct 31 01:14:19.662145 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:19.662103 4440 generic.go:356] "Generic (PLEG): container finished" podID="c80bb0f7-7c51-4344-91d9-a37dc7d21a83" containerID="09c52943d34c819ac6cddf84b089da9d4bbe5ae0929136054a66be43710e2ef7" exitCode=0 Oct 31 01:14:19.982410 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:19.662626 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-1b8d05-6f65d64999-htfv4" event={"ID":"c80bb0f7-7c51-4344-91d9-a37dc7d21a83","Type":"ContainerDied","Data":"09c52943d34c819ac6cddf84b089da9d4bbe5ae0929136054a66be43710e2ef7"} Oct 31 01:14:19.982410 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:19.757797 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" podUID="cf86081d-7942-454a-8763-f658bd0ce870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 31 01:14:19.982410 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:19.757830 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 31 01:14:19.982410 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:19.927925 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:19.982410 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:19.758536 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" podUID="2bde1a17-6626-433d-a82c-3cff6eac4a1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:19.982410 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:19.944850 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" podUID="2f72d419-0877-4e49-9a85-095ddea8c790" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:19.982410 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:19.949707 4440 generic.go:356] "Generic (PLEG): container finished" podID="7c3f1628-66ce-4761-80ea-e7654895d913" containerID="714f6ab78d14b43c7b0400742976aa34100b87b05b122814af1fb1b75090f16d" exitCode=0 Oct 31 01:14:20.289167 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.288959 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": read tcp 10.133.0.2:43798->10.133.0.68:8000: read: connection reset by peer" Oct 31 01:14:20.349460 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.317998 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1b8d05-5c48d56c8c-swf9d" event={"ID":"7c3f1628-66ce-4761-80ea-e7654895d913","Type":"ContainerDied","Data":"714f6ab78d14b43c7b0400742976aa34100b87b05b122814af1fb1b75090f16d"} Oct 31 01:14:20.357974 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.357939 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:20.358388 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.358344 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": dial tcp 10.133.0.68:8000: connect: connection refused" Oct 31 01:14:20.360305 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.360282 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx"] Oct 31 01:14:20.360565 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.360548 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" Oct 31 01:14:20.360845 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.360827 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" Oct 31 01:14:20.361001 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.360987 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d"] Oct 31 01:14:20.361136 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.361125 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq"] Oct 31 01:14:20.361310 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.361298 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd" Oct 31 01:14:20.363564 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.363531 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" podUID="961cf83c-7198-4f01-9a04-00d48bb61406" containerName="istio-proxy" containerID="cri-o://49ee265d5d7615a3f2f6d41e7ad9c12dbe984d1ff9a1b1cf834858150420f6b6" gracePeriod=30 Oct 31 01:14:20.364445 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.364206 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" podUID="81a46413-c078-447a-9bbb-d9ea9431469e" containerName="istio-proxy" containerID="cri-o://3b795f03e8a4813ba1a35e62abc282d37d77cc108e93b21613fd53c0d1dc8e41" gracePeriod=30 Oct 31 01:14:20.364445 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.364343 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:20.365103 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.365080 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" podUID="4eb48867-402c-40ca-8db3-c513636cbf20" containerName="istio-proxy" containerID="cri-o://524b582fa2d49e50204960dff4f1821661b4b193b9f2f4aec0a23d8183a3764b" gracePeriod=30 Oct 31 01:14:20.365258 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.365232 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 31 01:14:20.376190 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.375789 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" Oct 31 01:14:20.378316 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:14:20.378261 4440 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c3f1628_66ce_4761_80ea_e7654895d913.slice/crio-714f6ab78d14b43c7b0400742976aa34100b87b05b122814af1fb1b75090f16d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c3f1628_66ce_4761_80ea_e7654895d913.slice/crio-conmon-714f6ab78d14b43c7b0400742976aa34100b87b05b122814af1fb1b75090f16d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod617e420d_d029_4f4b_8ae5_63fe7d85971e.slice/crio-48f30cd271402c703cf948c7a6175c8b5b2778e52a098ec52cae25c96ee06d4b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc80bb0f7_7c51_4344_91d9_a37dc7d21a83.slice/crio-09c52943d34c819ac6cddf84b089da9d4bbe5ae0929136054a66be43710e2ef7.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod617e420d_d029_4f4b_8ae5_63fe7d85971e.slice/crio-conmon-48f30cd271402c703cf948c7a6175c8b5b2778e52a098ec52cae25c96ee06d4b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod478a9053_b8a9_4b70_a96d_aa50644aaf53.slice\": RecentStats: unable to find data in memory cache]" Oct 31 01:14:20.419774 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.402055 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": dial tcp 10.133.0.78:8000: connect: connection refused" Oct 31 01:14:20.419774 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.418795 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" Oct 31 01:14:20.511293 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:14:20.511004 4440 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ab080a7_963c_45db_a83c_2c359b1bcf06.slice/crio-conmon-e87b951688d5a10a0c118c8185589d2c6ac8eb73c32557d515181916a0cc1437.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2ca4946_eb55_4a6b_a4bb_257a5792ffed.slice/crio-7ab14572d848ec37a73de36151f934ccb2da98f4021a7dca143227067ade51be.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2ca4946_eb55_4a6b_a4bb_257a5792ffed.slice/crio-conmon-7ab14572d848ec37a73de36151f934ccb2da98f4021a7dca143227067ade51be.scope\": RecentStats: unable to find data in memory cache]" Oct 31 01:14:20.577687 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.576079 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:20.637910 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.637876 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-1b8d05-6f65d64999-htfv4" Oct 31 01:14:20.639856 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.639324 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" podUID="617e420d-d029-4f4b-8ae5-63fe7d85971e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": dial tcp 10.133.0.87:8000: connect: connection refused" Oct 31 01:14:20.640431 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.640396 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" Oct 31 01:14:20.673170 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.673139 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1b8d05-5c48d56c8c-swf9d" Oct 31 01:14:20.688880 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.688831 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": dial tcp 10.133.0.78:8000: connect: connection refused" Oct 31 01:14:20.710933 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.710709 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:20.713861 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.713831 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-28ff08-82b9g"] Oct 31 01:14:20.715201 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.715174 4440 cpu_manager.go:414] "containerMap: removing container" podUID="c80bb0f7-7c51-4344-91d9-a37dc7d21a83" containerName="payload-processing" Oct 31 01:14:20.715343 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.715330 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c80bb0f7-7c51-4344-91d9-a37dc7d21a83" containerName="payload-processing" Oct 31 01:14:20.715452 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.715441 4440 cpu_manager.go:414] "containerMap: removing container" podUID="7c3f1628-66ce-4761-80ea-e7654895d913" containerName="payload-pre-processing" Oct 31 01:14:20.715854 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.715836 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c3f1628-66ce-4761-80ea-e7654895d913" containerName="payload-pre-processing" Oct 31 01:14:20.715973 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.715962 4440 cpu_manager.go:414] "containerMap: removing container" podUID="478a9053-b8a9-4b70-a96d-aa50644aaf53" containerName="revoke-keys" Oct 31 01:14:20.716046 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.716035 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="478a9053-b8a9-4b70-a96d-aa50644aaf53" containerName="revoke-keys" Oct 31 01:14:20.716506 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.716490 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="c80bb0f7-7c51-4344-91d9-a37dc7d21a83" containerName="payload-processing" Oct 31 01:14:20.716610 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.716600 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c3f1628-66ce-4761-80ea-e7654895d913" containerName="payload-pre-processing" Oct 31 01:14:20.716663 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.716656 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="478a9053-b8a9-4b70-a96d-aa50644aaf53" containerName="revoke-keys" Oct 31 01:14:20.726783 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.726757 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-28ff08-82b9g" Oct 31 01:14:20.730388 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.730350 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-28ff08-82b9g"] Oct 31 01:14:20.731273 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.731252 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:14:20.731524 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.731510 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zs6l6\"" Oct 31 01:14:20.788157 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.788122 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c80bb0f7-7c51-4344-91d9-a37dc7d21a83-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80bb0f7-7c51-4344-91d9-a37dc7d21a83-plugins-config-volume\") pod \"c80bb0f7-7c51-4344-91d9-a37dc7d21a83\" (UID: \"c80bb0f7-7c51-4344-91d9-a37dc7d21a83\") " Oct 31 01:14:20.788688 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.788652 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c80bb0f7-7c51-4344-91d9-a37dc7d21a83-plugins-config-volume" pod "c80bb0f7-7c51-4344-91d9-a37dc7d21a83" (UID: "c80bb0f7-7c51-4344-91d9-a37dc7d21a83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:20.789576 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.789547 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c80bb0f7-7c51-4344-91d9-a37dc7d21a83-kube-api-access-k67tj\" (UniqueName: \"kubernetes.io/projected/c80bb0f7-7c51-4344-91d9-a37dc7d21a83-kube-api-access-k67tj\") pod \"c80bb0f7-7c51-4344-91d9-a37dc7d21a83\" (UID: \"c80bb0f7-7c51-4344-91d9-a37dc7d21a83\") " Oct 31 01:14:20.789788 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.789772 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c3f1628-66ce-4761-80ea-e7654895d913-kube-api-access-cgzqg\" (UniqueName: \"kubernetes.io/projected/7c3f1628-66ce-4761-80ea-e7654895d913-kube-api-access-cgzqg\") pod \"7c3f1628-66ce-4761-80ea-e7654895d913\" (UID: \"7c3f1628-66ce-4761-80ea-e7654895d913\") " Oct 31 01:14:20.789954 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.789940 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c3f1628-66ce-4761-80ea-e7654895d913-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c3f1628-66ce-4761-80ea-e7654895d913-plugins-config-volume\") pod \"7c3f1628-66ce-4761-80ea-e7654895d913\" (UID: \"7c3f1628-66ce-4761-80ea-e7654895d913\") " Oct 31 01:14:20.790289 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.790267 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b26d9f70-bea6-42e0-b5d2-c1b394501ddf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-28ff08-82b9g\" (UID: \"b26d9f70-bea6-42e0-b5d2-c1b394501ddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-28ff08-82b9g" Oct 31 01:14:20.790526 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.790512 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8x5k\" (UniqueName: \"kubernetes.io/projected/b26d9f70-bea6-42e0-b5d2-c1b394501ddf-kube-api-access-w8x5k\") pod \"maas-api-revoke-keys-e2e-worker-w5-28ff08-82b9g\" (UID: \"b26d9f70-bea6-42e0-b5d2-c1b394501ddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-28ff08-82b9g" Oct 31 01:14:20.790909 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.790891 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b26d9f70-bea6-42e0-b5d2-c1b394501ddf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-28ff08-82b9g\" (UID: \"b26d9f70-bea6-42e0-b5d2-c1b394501ddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-28ff08-82b9g" Oct 31 01:14:20.791134 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.791118 4440 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80bb0f7-7c51-4344-91d9-a37dc7d21a83-plugins-config-volume\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:20.791226 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.791187 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c3f1628-66ce-4761-80ea-e7654895d913-plugins-config-volume" pod "7c3f1628-66ce-4761-80ea-e7654895d913" (UID: "7c3f1628-66ce-4761-80ea-e7654895d913"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:20.796053 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.796006 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c80bb0f7-7c51-4344-91d9-a37dc7d21a83-kube-api-access-k67tj" pod "c80bb0f7-7c51-4344-91d9-a37dc7d21a83" (UID: "c80bb0f7-7c51-4344-91d9-a37dc7d21a83"). InnerVolumeSpecName "kube-api-access-k67tj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:20.804461 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.801006 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c3f1628-66ce-4761-80ea-e7654895d913-kube-api-access-cgzqg" pod "7c3f1628-66ce-4761-80ea-e7654895d913" (UID: "7c3f1628-66ce-4761-80ea-e7654895d913"). InnerVolumeSpecName "kube-api-access-cgzqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:20.868487 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.867640 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" Oct 31 01:14:20.893592 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.893525 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b26d9f70-bea6-42e0-b5d2-c1b394501ddf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-28ff08-82b9g\" (UID: \"b26d9f70-bea6-42e0-b5d2-c1b394501ddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-28ff08-82b9g" Oct 31 01:14:20.893755 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.893658 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b26d9f70-bea6-42e0-b5d2-c1b394501ddf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-28ff08-82b9g\" (UID: \"b26d9f70-bea6-42e0-b5d2-c1b394501ddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-28ff08-82b9g" Oct 31 01:14:20.893839 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.893793 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8x5k\" (UniqueName: \"kubernetes.io/projected/b26d9f70-bea6-42e0-b5d2-c1b394501ddf-kube-api-access-w8x5k\") pod \"maas-api-revoke-keys-e2e-worker-w5-28ff08-82b9g\" (UID: \"b26d9f70-bea6-42e0-b5d2-c1b394501ddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-28ff08-82b9g" Oct 31 01:14:20.895306 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.895272 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b26d9f70-bea6-42e0-b5d2-c1b394501ddf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-28ff08-82b9g\" (UID: \"b26d9f70-bea6-42e0-b5d2-c1b394501ddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-28ff08-82b9g" Oct 31 01:14:20.902473 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.902384 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k67tj\" (UniqueName: \"kubernetes.io/projected/c80bb0f7-7c51-4344-91d9-a37dc7d21a83-kube-api-access-k67tj\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:20.902727 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.902494 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgzqg\" (UniqueName: \"kubernetes.io/projected/7c3f1628-66ce-4761-80ea-e7654895d913-kube-api-access-cgzqg\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:20.902727 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.902583 4440 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c3f1628-66ce-4761-80ea-e7654895d913-plugins-config-volume\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:20.908188 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.908135 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b26d9f70-bea6-42e0-b5d2-c1b394501ddf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-28ff08-82b9g\" (UID: \"b26d9f70-bea6-42e0-b5d2-c1b394501ddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-28ff08-82b9g" Oct 31 01:14:20.913377 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.913345 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8x5k\" (UniqueName: \"kubernetes.io/projected/b26d9f70-bea6-42e0-b5d2-c1b394501ddf-kube-api-access-w8x5k\") pod \"maas-api-revoke-keys-e2e-worker-w5-28ff08-82b9g\" (UID: \"b26d9f70-bea6-42e0-b5d2-c1b394501ddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-28ff08-82b9g" Oct 31 01:14:20.992439 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.992382 4440 generic.go:356] "Generic (PLEG): container finished" podID="617e420d-d029-4f4b-8ae5-63fe7d85971e" containerID="48f30cd271402c703cf948c7a6175c8b5b2778e52a098ec52cae25c96ee06d4b" exitCode=0 Oct 31 01:14:20.992836 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.992810 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" event={"ID":"617e420d-d029-4f4b-8ae5-63fe7d85971e","Type":"ContainerDied","Data":"48f30cd271402c703cf948c7a6175c8b5b2778e52a098ec52cae25c96ee06d4b"} Oct 31 01:14:20.993077 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.993049 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" event={"ID":"617e420d-d029-4f4b-8ae5-63fe7d85971e","Type":"ContainerDied","Data":"5ab066f043a74434317766b4e16a2aed8b5f7820ec60e46c36a2f261f101bb2d"} Oct 31 01:14:20.993207 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:20.993180 4440 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ab066f043a74434317766b4e16a2aed8b5f7820ec60e46c36a2f261f101bb2d" Oct 31 01:14:21.023843 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.004257 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-1b8d05-6f65d64999-htfv4" event={"ID":"c80bb0f7-7c51-4344-91d9-a37dc7d21a83","Type":"ContainerDied","Data":"a82e12f53fd7789b48fc61cb4fed8384fbd5bb69920514a6be18b7ebcd59917c"} Oct 31 01:14:21.023843 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.004341 4440 scope.go:122] "RemoveContainer" containerID="09c52943d34c819ac6cddf84b089da9d4bbe5ae0929136054a66be43710e2ef7" Oct 31 01:14:21.023843 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.004608 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-1b8d05-6f65d64999-htfv4" Oct 31 01:14:21.023843 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.007638 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ab080a7-963c-45db-a83c-2c359b1bcf06-kube-api-access-pqgtr\" (UniqueName: \"kubernetes.io/projected/4ab080a7-963c-45db-a83c-2c359b1bcf06-kube-api-access-pqgtr\") pod \"4ab080a7-963c-45db-a83c-2c359b1bcf06\" (UID: \"4ab080a7-963c-45db-a83c-2c359b1bcf06\") " Oct 31 01:14:21.023843 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.007723 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ab080a7-963c-45db-a83c-2c359b1bcf06-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ab080a7-963c-45db-a83c-2c359b1bcf06-model-cache\") pod \"4ab080a7-963c-45db-a83c-2c359b1bcf06\" (UID: \"4ab080a7-963c-45db-a83c-2c359b1bcf06\") " Oct 31 01:14:21.023843 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.007805 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ab080a7-963c-45db-a83c-2c359b1bcf06-tls-certs\" (UniqueName: \"kubernetes.io/secret/4ab080a7-963c-45db-a83c-2c359b1bcf06-tls-certs\") pod \"4ab080a7-963c-45db-a83c-2c359b1bcf06\" (UID: \"4ab080a7-963c-45db-a83c-2c359b1bcf06\") " Oct 31 01:14:21.023843 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.007848 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ab080a7-963c-45db-a83c-2c359b1bcf06-home\" (UniqueName: \"kubernetes.io/empty-dir/4ab080a7-963c-45db-a83c-2c359b1bcf06-home\") pod \"4ab080a7-963c-45db-a83c-2c359b1bcf06\" (UID: \"4ab080a7-963c-45db-a83c-2c359b1bcf06\") " Oct 31 01:14:21.023843 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.007949 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ab080a7-963c-45db-a83c-2c359b1bcf06-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ab080a7-963c-45db-a83c-2c359b1bcf06-tmp-dir\") pod \"4ab080a7-963c-45db-a83c-2c359b1bcf06\" (UID: \"4ab080a7-963c-45db-a83c-2c359b1bcf06\") " Oct 31 01:14:21.023843 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.008047 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ab080a7-963c-45db-a83c-2c359b1bcf06-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ab080a7-963c-45db-a83c-2c359b1bcf06-dshm\") pod \"4ab080a7-963c-45db-a83c-2c359b1bcf06\" (UID: \"4ab080a7-963c-45db-a83c-2c359b1bcf06\") " Oct 31 01:14:21.023843 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.011744 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ab080a7-963c-45db-a83c-2c359b1bcf06-dshm" pod "4ab080a7-963c-45db-a83c-2c359b1bcf06" (UID: "4ab080a7-963c-45db-a83c-2c359b1bcf06"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:21.023843 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.012409 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ab080a7-963c-45db-a83c-2c359b1bcf06-home" pod "4ab080a7-963c-45db-a83c-2c359b1bcf06" (UID: "4ab080a7-963c-45db-a83c-2c359b1bcf06"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:21.023843 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.012601 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ab080a7-963c-45db-a83c-2c359b1bcf06-tmp-dir" pod "4ab080a7-963c-45db-a83c-2c359b1bcf06" (UID: "4ab080a7-963c-45db-a83c-2c359b1bcf06"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:21.023843 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.013635 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ab080a7-963c-45db-a83c-2c359b1bcf06-kube-api-access-pqgtr" pod "4ab080a7-963c-45db-a83c-2c359b1bcf06" (UID: "4ab080a7-963c-45db-a83c-2c359b1bcf06"). InnerVolumeSpecName "kube-api-access-pqgtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:21.023843 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.013726 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ab080a7-963c-45db-a83c-2c359b1bcf06-model-cache" pod "4ab080a7-963c-45db-a83c-2c359b1bcf06" (UID: "4ab080a7-963c-45db-a83c-2c359b1bcf06"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:21.023843 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.014457 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ab080a7-963c-45db-a83c-2c359b1bcf06-tls-certs" pod "4ab080a7-963c-45db-a83c-2c359b1bcf06" (UID: "4ab080a7-963c-45db-a83c-2c359b1bcf06"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:14:21.023843 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.016029 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1b8d05-5c48d56c8c-swf9d" event={"ID":"7c3f1628-66ce-4761-80ea-e7654895d913","Type":"ContainerDied","Data":"4e2d29b911ee825cd3960936aea2135eb9e12328bd9db595f2fd693419c5fa17"} Oct 31 01:14:21.023843 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.016140 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1b8d05-5c48d56c8c-swf9d" Oct 31 01:14:21.028028 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.027789 4440 generic.go:356] "Generic (PLEG): container finished" podID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerID="7ab14572d848ec37a73de36151f934ccb2da98f4021a7dca143227067ade51be" exitCode=0 Oct 31 01:14:21.028028 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.027950 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" event={"ID":"f2ca4946-eb55-4a6b-a4bb-257a5792ffed","Type":"ContainerDied","Data":"7ab14572d848ec37a73de36151f934ccb2da98f4021a7dca143227067ade51be"} Oct 31 01:14:21.051911 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.051856 4440 generic.go:356] "Generic (PLEG): container finished" podID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerID="e87b951688d5a10a0c118c8185589d2c6ac8eb73c32557d515181916a0cc1437" exitCode=0 Oct 31 01:14:21.053874 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.052034 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" event={"ID":"4ab080a7-963c-45db-a83c-2c359b1bcf06","Type":"ContainerDied","Data":"e87b951688d5a10a0c118c8185589d2c6ac8eb73c32557d515181916a0cc1437"} Oct 31 01:14:21.053874 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.052100 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" event={"ID":"4ab080a7-963c-45db-a83c-2c359b1bcf06","Type":"ContainerDied","Data":"f454110f5084afe957195d08ceb0e766f574bdb27a969d71a5f790525d99d43d"} Oct 31 01:14:21.053874 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.052185 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq" Oct 31 01:14:21.065776 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.065747 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-28ff08-82b9g" Oct 31 01:14:21.071136 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.071049 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" Oct 31 01:14:21.074635 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.074004 4440 scope.go:122] "RemoveContainer" containerID="714f6ab78d14b43c7b0400742976aa34100b87b05b122814af1fb1b75090f16d" Oct 31 01:14:21.080282 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.080256 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" Oct 31 01:14:21.089570 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.089385 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-1b8d05-5c48d56c8c-swf9d"] Oct 31 01:14:21.089948 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.089923 4440 scope.go:122] "RemoveContainer" containerID="e87b951688d5a10a0c118c8185589d2c6ac8eb73c32557d515181916a0cc1437" Oct 31 01:14:21.094784 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.094758 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-1b8d05-5c48d56c8c-swf9d"] Oct 31 01:14:21.106406 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.106329 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq"] Oct 31 01:14:21.120624 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.110297 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2ca4946-eb55-4a6b-a4bb-257a5792ffed-home\" (UniqueName: \"kubernetes.io/empty-dir/f2ca4946-eb55-4a6b-a4bb-257a5792ffed-home\") pod \"f2ca4946-eb55-4a6b-a4bb-257a5792ffed\" (UID: \"f2ca4946-eb55-4a6b-a4bb-257a5792ffed\") " Oct 31 01:14:21.120624 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.110483 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2ca4946-eb55-4a6b-a4bb-257a5792ffed-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2ca4946-eb55-4a6b-a4bb-257a5792ffed-model-cache\") pod \"f2ca4946-eb55-4a6b-a4bb-257a5792ffed\" (UID: \"f2ca4946-eb55-4a6b-a4bb-257a5792ffed\") " Oct 31 01:14:21.120624 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.110524 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2ca4946-eb55-4a6b-a4bb-257a5792ffed-kube-api-access-cfj7l\" (UniqueName: \"kubernetes.io/projected/f2ca4946-eb55-4a6b-a4bb-257a5792ffed-kube-api-access-cfj7l\") pod \"f2ca4946-eb55-4a6b-a4bb-257a5792ffed\" (UID: \"f2ca4946-eb55-4a6b-a4bb-257a5792ffed\") " Oct 31 01:14:21.120624 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.110581 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/617e420d-d029-4f4b-8ae5-63fe7d85971e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/617e420d-d029-4f4b-8ae5-63fe7d85971e-dshm\") pod \"617e420d-d029-4f4b-8ae5-63fe7d85971e\" (UID: \"617e420d-d029-4f4b-8ae5-63fe7d85971e\") " Oct 31 01:14:21.120624 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.110625 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/617e420d-d029-4f4b-8ae5-63fe7d85971e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/617e420d-d029-4f4b-8ae5-63fe7d85971e-tmp-dir\") pod \"617e420d-d029-4f4b-8ae5-63fe7d85971e\" (UID: \"617e420d-d029-4f4b-8ae5-63fe7d85971e\") " Oct 31 01:14:21.120624 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.110701 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/617e420d-d029-4f4b-8ae5-63fe7d85971e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/617e420d-d029-4f4b-8ae5-63fe7d85971e-model-cache\") pod \"617e420d-d029-4f4b-8ae5-63fe7d85971e\" (UID: \"617e420d-d029-4f4b-8ae5-63fe7d85971e\") " Oct 31 01:14:21.120624 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.110742 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/617e420d-d029-4f4b-8ae5-63fe7d85971e-home\" (UniqueName: \"kubernetes.io/empty-dir/617e420d-d029-4f4b-8ae5-63fe7d85971e-home\") pod \"617e420d-d029-4f4b-8ae5-63fe7d85971e\" (UID: \"617e420d-d029-4f4b-8ae5-63fe7d85971e\") " Oct 31 01:14:21.120624 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.110779 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2ca4946-eb55-4a6b-a4bb-257a5792ffed-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2ca4946-eb55-4a6b-a4bb-257a5792ffed-dshm\") pod \"f2ca4946-eb55-4a6b-a4bb-257a5792ffed\" (UID: \"f2ca4946-eb55-4a6b-a4bb-257a5792ffed\") " Oct 31 01:14:21.120624 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.110933 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/617e420d-d029-4f4b-8ae5-63fe7d85971e-tls-certs\" (UniqueName: \"kubernetes.io/secret/617e420d-d029-4f4b-8ae5-63fe7d85971e-tls-certs\") pod \"617e420d-d029-4f4b-8ae5-63fe7d85971e\" (UID: \"617e420d-d029-4f4b-8ae5-63fe7d85971e\") " Oct 31 01:14:21.120624 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.110977 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2ca4946-eb55-4a6b-a4bb-257a5792ffed-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2ca4946-eb55-4a6b-a4bb-257a5792ffed-tls-certs\") pod \"f2ca4946-eb55-4a6b-a4bb-257a5792ffed\" (UID: \"f2ca4946-eb55-4a6b-a4bb-257a5792ffed\") " Oct 31 01:14:21.120624 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.111022 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2ca4946-eb55-4a6b-a4bb-257a5792ffed-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2ca4946-eb55-4a6b-a4bb-257a5792ffed-tmp-dir\") pod \"f2ca4946-eb55-4a6b-a4bb-257a5792ffed\" (UID: \"f2ca4946-eb55-4a6b-a4bb-257a5792ffed\") " Oct 31 01:14:21.120624 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.111072 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/617e420d-d029-4f4b-8ae5-63fe7d85971e-kube-api-access-t9c4h\" (UniqueName: \"kubernetes.io/projected/617e420d-d029-4f4b-8ae5-63fe7d85971e-kube-api-access-t9c4h\") pod \"617e420d-d029-4f4b-8ae5-63fe7d85971e\" (UID: \"617e420d-d029-4f4b-8ae5-63fe7d85971e\") " Oct 31 01:14:21.120624 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.111839 4440 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ab080a7-963c-45db-a83c-2c359b1bcf06-dshm\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:21.120624 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.111863 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqgtr\" (UniqueName: \"kubernetes.io/projected/4ab080a7-963c-45db-a83c-2c359b1bcf06-kube-api-access-pqgtr\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:21.120624 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.111879 4440 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ab080a7-963c-45db-a83c-2c359b1bcf06-model-cache\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:21.120624 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.111893 4440 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ab080a7-963c-45db-a83c-2c359b1bcf06-tls-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:21.120624 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.111906 4440 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ab080a7-963c-45db-a83c-2c359b1bcf06-home\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:21.120624 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.111918 4440 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ab080a7-963c-45db-a83c-2c359b1bcf06-tmp-dir\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:21.120624 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.112842 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-0a6cca/simulator-w2-0a6cca-kserve-7cb6bc9dd9-fglkq"] Oct 31 01:14:21.120624 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.113169 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/617e420d-d029-4f4b-8ae5-63fe7d85971e-model-cache" pod "617e420d-d029-4f4b-8ae5-63fe7d85971e" (UID: "617e420d-d029-4f4b-8ae5-63fe7d85971e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:21.120624 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.113371 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ca4946-eb55-4a6b-a4bb-257a5792ffed-home" pod "f2ca4946-eb55-4a6b-a4bb-257a5792ffed" (UID: "f2ca4946-eb55-4a6b-a4bb-257a5792ffed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:21.120624 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.113569 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ca4946-eb55-4a6b-a4bb-257a5792ffed-model-cache" pod "f2ca4946-eb55-4a6b-a4bb-257a5792ffed" (UID: "f2ca4946-eb55-4a6b-a4bb-257a5792ffed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:21.120624 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.116513 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2ca4946-eb55-4a6b-a4bb-257a5792ffed-kube-api-access-cfj7l" pod "f2ca4946-eb55-4a6b-a4bb-257a5792ffed" (UID: "f2ca4946-eb55-4a6b-a4bb-257a5792ffed"). InnerVolumeSpecName "kube-api-access-cfj7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:21.120624 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.119716 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/617e420d-d029-4f4b-8ae5-63fe7d85971e-dshm" pod "617e420d-d029-4f4b-8ae5-63fe7d85971e" (UID: "617e420d-d029-4f4b-8ae5-63fe7d85971e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:21.120624 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.120044 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/617e420d-d029-4f4b-8ae5-63fe7d85971e-tmp-dir" pod "617e420d-d029-4f4b-8ae5-63fe7d85971e" (UID: "617e420d-d029-4f4b-8ae5-63fe7d85971e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:21.126212 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.125900 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/617e420d-d029-4f4b-8ae5-63fe7d85971e-home" pod "617e420d-d029-4f4b-8ae5-63fe7d85971e" (UID: "617e420d-d029-4f4b-8ae5-63fe7d85971e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:21.127814 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.127194 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/617e420d-d029-4f4b-8ae5-63fe7d85971e-kube-api-access-t9c4h" pod "617e420d-d029-4f4b-8ae5-63fe7d85971e" (UID: "617e420d-d029-4f4b-8ae5-63fe7d85971e"). InnerVolumeSpecName "kube-api-access-t9c4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:21.127814 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.127542 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ca4946-eb55-4a6b-a4bb-257a5792ffed-tmp-dir" pod "f2ca4946-eb55-4a6b-a4bb-257a5792ffed" (UID: "f2ca4946-eb55-4a6b-a4bb-257a5792ffed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:21.133056 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.132762 4440 scope.go:122] "RemoveContainer" containerID="e87b951688d5a10a0c118c8185589d2c6ac8eb73c32557d515181916a0cc1437" Oct 31 01:14:21.133402 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.133332 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/617e420d-d029-4f4b-8ae5-63fe7d85971e-tls-certs" pod "617e420d-d029-4f4b-8ae5-63fe7d85971e" (UID: "617e420d-d029-4f4b-8ae5-63fe7d85971e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:14:21.133836 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.133812 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ca4946-eb55-4a6b-a4bb-257a5792ffed-dshm" pod "f2ca4946-eb55-4a6b-a4bb-257a5792ffed" (UID: "f2ca4946-eb55-4a6b-a4bb-257a5792ffed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:21.134815 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.134780 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2ca4946-eb55-4a6b-a4bb-257a5792ffed-tls-certs" pod "f2ca4946-eb55-4a6b-a4bb-257a5792ffed" (UID: "f2ca4946-eb55-4a6b-a4bb-257a5792ffed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:14:21.137841 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:14:21.137666 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e87b951688d5a10a0c118c8185589d2c6ac8eb73c32557d515181916a0cc1437\": container with ID starting with e87b951688d5a10a0c118c8185589d2c6ac8eb73c32557d515181916a0cc1437 not found: ID does not exist" containerID="e87b951688d5a10a0c118c8185589d2c6ac8eb73c32557d515181916a0cc1437" Oct 31 01:14:21.137841 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.137714 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e87b951688d5a10a0c118c8185589d2c6ac8eb73c32557d515181916a0cc1437"} err="failed to get container status \"e87b951688d5a10a0c118c8185589d2c6ac8eb73c32557d515181916a0cc1437\": rpc error: code = NotFound desc = could not find container \"e87b951688d5a10a0c118c8185589d2c6ac8eb73c32557d515181916a0cc1437\": container with ID starting with e87b951688d5a10a0c118c8185589d2c6ac8eb73c32557d515181916a0cc1437 not found: ID does not exist" Oct 31 01:14:21.210837 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.210764 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-1b8d05-6f65d64999-htfv4"] Oct 31 01:14:21.218499 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.218464 4440 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/617e420d-d029-4f4b-8ae5-63fe7d85971e-dshm\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:21.218736 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.218721 4440 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/617e420d-d029-4f4b-8ae5-63fe7d85971e-tmp-dir\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:21.218835 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.218823 4440 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/617e420d-d029-4f4b-8ae5-63fe7d85971e-model-cache\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:21.218914 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.218903 4440 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/617e420d-d029-4f4b-8ae5-63fe7d85971e-home\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:21.218990 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.218978 4440 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2ca4946-eb55-4a6b-a4bb-257a5792ffed-dshm\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:21.219076 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.219065 4440 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/617e420d-d029-4f4b-8ae5-63fe7d85971e-tls-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:21.219159 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.219150 4440 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2ca4946-eb55-4a6b-a4bb-257a5792ffed-tls-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:21.219229 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.219219 4440 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2ca4946-eb55-4a6b-a4bb-257a5792ffed-tmp-dir\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:21.219309 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.219296 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9c4h\" (UniqueName: \"kubernetes.io/projected/617e420d-d029-4f4b-8ae5-63fe7d85971e-kube-api-access-t9c4h\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:21.219392 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.219379 4440 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2ca4946-eb55-4a6b-a4bb-257a5792ffed-home\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:21.219489 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.219478 4440 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2ca4946-eb55-4a6b-a4bb-257a5792ffed-model-cache\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:21.219570 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.219558 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfj7l\" (UniqueName: \"kubernetes.io/projected/f2ca4946-eb55-4a6b-a4bb-257a5792ffed-kube-api-access-cfj7l\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:21.219648 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.219615 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-1b8d05-6f65d64999-htfv4"] Oct 31 01:14:21.390074 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.390038 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-28ff08-82b9g"] Oct 31 01:14:21.390750 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:14:21.390718 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb26d9f70_bea6_42e0_b5d2_c1b394501ddf.slice/crio-d16869d443d751783c1716477f019ba8901515638e9a0ca320ca7e0c8c2c65d4 WatchSource:0}: Error finding container d16869d443d751783c1716477f019ba8901515638e9a0ca320ca7e0c8c2c65d4: Status 404 returned error can't find the container with id d16869d443d751783c1716477f019ba8901515638e9a0ca320ca7e0c8c2c65d4 Oct 31 01:14:21.773520 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.773483 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" path="/var/lib/kubelet/pods/4ab080a7-963c-45db-a83c-2c359b1bcf06/volumes" Oct 31 01:14:21.774035 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.774020 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c3f1628-66ce-4761-80ea-e7654895d913" path="/var/lib/kubelet/pods/7c3f1628-66ce-4761-80ea-e7654895d913/volumes" Oct 31 01:14:21.774481 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.774467 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c80bb0f7-7c51-4344-91d9-a37dc7d21a83" path="/var/lib/kubelet/pods/c80bb0f7-7c51-4344-91d9-a37dc7d21a83/volumes" Oct 31 01:14:21.809876 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.809840 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:21.815761 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:21.815708 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8499c8-5c65ddd74c-9r5n8" podUID="ab8ba016-584a-4646-a3d4-f6db4ccf0700" containerName="payload-pre-processing" containerID="cri-o://ba5565b2f2730d579c090408859c176f458b63d24c3adf03db39fe85462fe15e" gracePeriod=25 Oct 31 01:14:22.064264 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:22.064179 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" Oct 31 01:14:22.064491 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:22.064273 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv" event={"ID":"f2ca4946-eb55-4a6b-a4bb-257a5792ffed","Type":"ContainerDied","Data":"7746e8bedb4d1977ac24b0f2e11d87eed164e32e112584ccab7ca498c2587877"} Oct 31 01:14:22.064491 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:22.064332 4440 scope.go:122] "RemoveContainer" containerID="7ab14572d848ec37a73de36151f934ccb2da98f4021a7dca143227067ade51be" Oct 31 01:14:22.066105 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:22.066073 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-28ff08-82b9g" event={"ID":"b26d9f70-bea6-42e0-b5d2-c1b394501ddf","Type":"ContainerStarted","Data":"c4555074111df94cf53b0eb4773405356efb2bf0d670e4c2f82a4f3af6b1a847"} Oct 31 01:14:22.066234 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:22.066142 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-28ff08-82b9g" event={"ID":"b26d9f70-bea6-42e0-b5d2-c1b394501ddf","Type":"ContainerStarted","Data":"d16869d443d751783c1716477f019ba8901515638e9a0ca320ca7e0c8c2c65d4"} Oct 31 01:14:22.067593 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:22.067567 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4" Oct 31 01:14:22.105182 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:22.105138 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv"] Oct 31 01:14:22.113052 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:22.113010 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-0a6cca/distinct-w2-0a6cca-kserve-667d9c7c99-rtxgv"] Oct 31 01:14:22.132932 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:22.132859 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-28ff08-82b9g" podStartSLOduration=2.132828952 podStartE2EDuration="2.132828952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:14:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:14:22.113838084 +0000 UTC m=+1106.472560421" watchObservedRunningTime="2026-10-31 01:14:22.132828952 +0000 UTC m=+1106.491551267" Oct 31 01:14:22.134920 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:22.134863 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4"] Oct 31 01:14:22.139723 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:22.139692 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-0a6cca/distinct-2-w2-0a6cca-kserve-77868965c6-69cv4"] Oct 31 01:14:22.312153 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:22.312111 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:22.463443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:22.459875 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:22.913240 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:22.913150 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:23.081085 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:23.081047 4440 generic.go:356] "Generic (PLEG): container finished" podID="b26d9f70-bea6-42e0-b5d2-c1b394501ddf" containerID="c4555074111df94cf53b0eb4773405356efb2bf0d670e4c2f82a4f3af6b1a847" exitCode=0 Oct 31 01:14:23.081257 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:23.081131 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-28ff08-82b9g" event={"ID":"b26d9f70-bea6-42e0-b5d2-c1b394501ddf","Type":"ContainerDied","Data":"c4555074111df94cf53b0eb4773405356efb2bf0d670e4c2f82a4f3af6b1a847"} Oct 31 01:14:23.176248 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:23.176210 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:23.526508 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:23.519165 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:23.775684 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:23.775648 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="617e420d-d029-4f4b-8ae5-63fe7d85971e" path="/var/lib/kubelet/pods/617e420d-d029-4f4b-8ae5-63fe7d85971e/volumes" Oct 31 01:14:23.776312 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:23.776283 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" path="/var/lib/kubelet/pods/f2ca4946-eb55-4a6b-a4bb-257a5792ffed/volumes" Oct 31 01:14:23.836233 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:23.836139 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" podUID="2f72d419-0877-4e49-9a85-095ddea8c790" containerName="main" containerID="cri-o://2379083646d1c3ced10003652094f928ee54c73768e44bb54a990a7dd059da98" gracePeriod=45 Oct 31 01:14:23.855208 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:23.855173 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" podUID="0b82b50e-bfa7-45b9-8484-c7a331b5ac03" containerName="main" containerID="cri-o://d7b84ec0213b0cea20e0bb3b852fa1cecec5bbda6f4d8501bc5735874050bed8" gracePeriod=45 Oct 31 01:14:24.062257 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.061551 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:24.095432 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.095337 4440 generic.go:356] "Generic (PLEG): container finished" podID="0b82b50e-bfa7-45b9-8484-c7a331b5ac03" containerID="d7b84ec0213b0cea20e0bb3b852fa1cecec5bbda6f4d8501bc5735874050bed8" exitCode=0 Oct 31 01:14:24.095850 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.095817 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" event={"ID":"0b82b50e-bfa7-45b9-8484-c7a331b5ac03","Type":"ContainerDied","Data":"d7b84ec0213b0cea20e0bb3b852fa1cecec5bbda6f4d8501bc5735874050bed8"} Oct 31 01:14:24.098228 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.098204 4440 generic.go:356] "Generic (PLEG): container finished" podID="2f72d419-0877-4e49-9a85-095ddea8c790" containerID="2379083646d1c3ced10003652094f928ee54c73768e44bb54a990a7dd059da98" exitCode=0 Oct 31 01:14:24.098662 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.098637 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" event={"ID":"2f72d419-0877-4e49-9a85-095ddea8c790","Type":"ContainerDied","Data":"2379083646d1c3ced10003652094f928ee54c73768e44bb54a990a7dd059da98"} Oct 31 01:14:24.182495 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.182456 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:24.335757 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.335086 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" Oct 31 01:14:24.347226 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.347144 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" Oct 31 01:14:24.347639 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.347518 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-0a6cca-6599d4945f-dlflj" podUID="eaad3e7d-1293-43d5-9fe6-674e69ea840e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:14:24.409078 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.409038 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b82b50e-bfa7-45b9-8484-c7a331b5ac03-tls-certs\" (UniqueName: \"kubernetes.io/secret/0b82b50e-bfa7-45b9-8484-c7a331b5ac03-tls-certs\") pod \"0b82b50e-bfa7-45b9-8484-c7a331b5ac03\" (UID: \"0b82b50e-bfa7-45b9-8484-c7a331b5ac03\") " Oct 31 01:14:24.409265 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.409108 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b82b50e-bfa7-45b9-8484-c7a331b5ac03-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b82b50e-bfa7-45b9-8484-c7a331b5ac03-dshm\") pod \"0b82b50e-bfa7-45b9-8484-c7a331b5ac03\" (UID: \"0b82b50e-bfa7-45b9-8484-c7a331b5ac03\") " Oct 31 01:14:24.409265 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.409155 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f72d419-0877-4e49-9a85-095ddea8c790-kube-api-access-rfz49\" (UniqueName: \"kubernetes.io/projected/2f72d419-0877-4e49-9a85-095ddea8c790-kube-api-access-rfz49\") pod \"2f72d419-0877-4e49-9a85-095ddea8c790\" (UID: \"2f72d419-0877-4e49-9a85-095ddea8c790\") " Oct 31 01:14:24.409265 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.409182 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f72d419-0877-4e49-9a85-095ddea8c790-tls-certs\" (UniqueName: \"kubernetes.io/secret/2f72d419-0877-4e49-9a85-095ddea8c790-tls-certs\") pod \"2f72d419-0877-4e49-9a85-095ddea8c790\" (UID: \"2f72d419-0877-4e49-9a85-095ddea8c790\") " Oct 31 01:14:24.409265 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.409230 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f72d419-0877-4e49-9a85-095ddea8c790-home\" (UniqueName: \"kubernetes.io/empty-dir/2f72d419-0877-4e49-9a85-095ddea8c790-home\") pod \"2f72d419-0877-4e49-9a85-095ddea8c790\" (UID: \"2f72d419-0877-4e49-9a85-095ddea8c790\") " Oct 31 01:14:24.409265 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.409262 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b82b50e-bfa7-45b9-8484-c7a331b5ac03-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b82b50e-bfa7-45b9-8484-c7a331b5ac03-tmp-dir\") pod \"0b82b50e-bfa7-45b9-8484-c7a331b5ac03\" (UID: \"0b82b50e-bfa7-45b9-8484-c7a331b5ac03\") " Oct 31 01:14:24.409544 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.409333 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f72d419-0877-4e49-9a85-095ddea8c790-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f72d419-0877-4e49-9a85-095ddea8c790-dshm\") pod \"2f72d419-0877-4e49-9a85-095ddea8c790\" (UID: \"2f72d419-0877-4e49-9a85-095ddea8c790\") " Oct 31 01:14:24.409544 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.409456 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b82b50e-bfa7-45b9-8484-c7a331b5ac03-kube-api-access-t2qmr\" (UniqueName: \"kubernetes.io/projected/0b82b50e-bfa7-45b9-8484-c7a331b5ac03-kube-api-access-t2qmr\") pod \"0b82b50e-bfa7-45b9-8484-c7a331b5ac03\" (UID: \"0b82b50e-bfa7-45b9-8484-c7a331b5ac03\") " Oct 31 01:14:24.409544 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.409500 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f72d419-0877-4e49-9a85-095ddea8c790-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f72d419-0877-4e49-9a85-095ddea8c790-tmp-dir\") pod \"2f72d419-0877-4e49-9a85-095ddea8c790\" (UID: \"2f72d419-0877-4e49-9a85-095ddea8c790\") " Oct 31 01:14:24.409722 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.409584 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f72d419-0877-4e49-9a85-095ddea8c790-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f72d419-0877-4e49-9a85-095ddea8c790-model-cache\") pod \"2f72d419-0877-4e49-9a85-095ddea8c790\" (UID: \"2f72d419-0877-4e49-9a85-095ddea8c790\") " Oct 31 01:14:24.409722 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.409646 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b82b50e-bfa7-45b9-8484-c7a331b5ac03-home\" (UniqueName: \"kubernetes.io/empty-dir/0b82b50e-bfa7-45b9-8484-c7a331b5ac03-home\") pod \"0b82b50e-bfa7-45b9-8484-c7a331b5ac03\" (UID: \"0b82b50e-bfa7-45b9-8484-c7a331b5ac03\") " Oct 31 01:14:24.409722 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.409719 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b82b50e-bfa7-45b9-8484-c7a331b5ac03-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b82b50e-bfa7-45b9-8484-c7a331b5ac03-model-cache\") pod \"0b82b50e-bfa7-45b9-8484-c7a331b5ac03\" (UID: \"0b82b50e-bfa7-45b9-8484-c7a331b5ac03\") " Oct 31 01:14:24.410831 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.410797 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b82b50e-bfa7-45b9-8484-c7a331b5ac03-model-cache" pod "0b82b50e-bfa7-45b9-8484-c7a331b5ac03" (UID: "0b82b50e-bfa7-45b9-8484-c7a331b5ac03"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:24.412946 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.412919 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b82b50e-bfa7-45b9-8484-c7a331b5ac03-tmp-dir" pod "0b82b50e-bfa7-45b9-8484-c7a331b5ac03" (UID: "0b82b50e-bfa7-45b9-8484-c7a331b5ac03"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:24.413261 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.413239 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f72d419-0877-4e49-9a85-095ddea8c790-tmp-dir" pod "2f72d419-0877-4e49-9a85-095ddea8c790" (UID: "2f72d419-0877-4e49-9a85-095ddea8c790"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:24.413347 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.413255 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f72d419-0877-4e49-9a85-095ddea8c790-model-cache" pod "2f72d419-0877-4e49-9a85-095ddea8c790" (UID: "2f72d419-0877-4e49-9a85-095ddea8c790"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:24.413807 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.413759 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b82b50e-bfa7-45b9-8484-c7a331b5ac03-home" pod "0b82b50e-bfa7-45b9-8484-c7a331b5ac03" (UID: "0b82b50e-bfa7-45b9-8484-c7a331b5ac03"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:24.414032 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.413991 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f72d419-0877-4e49-9a85-095ddea8c790-home" pod "2f72d419-0877-4e49-9a85-095ddea8c790" (UID: "2f72d419-0877-4e49-9a85-095ddea8c790"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:24.415916 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.415895 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b82b50e-bfa7-45b9-8484-c7a331b5ac03-tls-certs" pod "0b82b50e-bfa7-45b9-8484-c7a331b5ac03" (UID: "0b82b50e-bfa7-45b9-8484-c7a331b5ac03"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:14:24.417161 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.417133 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f72d419-0877-4e49-9a85-095ddea8c790-tls-certs" pod "2f72d419-0877-4e49-9a85-095ddea8c790" (UID: "2f72d419-0877-4e49-9a85-095ddea8c790"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:14:24.418669 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.418644 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b82b50e-bfa7-45b9-8484-c7a331b5ac03-dshm" pod "0b82b50e-bfa7-45b9-8484-c7a331b5ac03" (UID: "0b82b50e-bfa7-45b9-8484-c7a331b5ac03"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:24.418908 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.418884 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f72d419-0877-4e49-9a85-095ddea8c790-kube-api-access-rfz49" pod "2f72d419-0877-4e49-9a85-095ddea8c790" (UID: "2f72d419-0877-4e49-9a85-095ddea8c790"). InnerVolumeSpecName "kube-api-access-rfz49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:24.419196 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.419166 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f72d419-0877-4e49-9a85-095ddea8c790-dshm" pod "2f72d419-0877-4e49-9a85-095ddea8c790" (UID: "2f72d419-0877-4e49-9a85-095ddea8c790"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:24.419326 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.419304 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b82b50e-bfa7-45b9-8484-c7a331b5ac03-kube-api-access-t2qmr" pod "0b82b50e-bfa7-45b9-8484-c7a331b5ac03" (UID: "0b82b50e-bfa7-45b9-8484-c7a331b5ac03"). InnerVolumeSpecName "kube-api-access-t2qmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:24.462777 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.462739 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-28ff08-82b9g" Oct 31 01:14:24.512820 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.512772 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2qmr\" (UniqueName: \"kubernetes.io/projected/0b82b50e-bfa7-45b9-8484-c7a331b5ac03-kube-api-access-t2qmr\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:24.512820 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.512822 4440 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f72d419-0877-4e49-9a85-095ddea8c790-tmp-dir\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:24.513099 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.512840 4440 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f72d419-0877-4e49-9a85-095ddea8c790-model-cache\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:24.513099 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.512855 4440 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b82b50e-bfa7-45b9-8484-c7a331b5ac03-home\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:24.513099 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.512868 4440 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b82b50e-bfa7-45b9-8484-c7a331b5ac03-model-cache\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:24.513099 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.512882 4440 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b82b50e-bfa7-45b9-8484-c7a331b5ac03-tls-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:24.513099 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.512895 4440 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b82b50e-bfa7-45b9-8484-c7a331b5ac03-dshm\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:24.513099 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.512909 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfz49\" (UniqueName: \"kubernetes.io/projected/2f72d419-0877-4e49-9a85-095ddea8c790-kube-api-access-rfz49\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:24.513099 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.512924 4440 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f72d419-0877-4e49-9a85-095ddea8c790-tls-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:24.513099 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.512937 4440 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f72d419-0877-4e49-9a85-095ddea8c790-home\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:24.513099 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.512950 4440 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b82b50e-bfa7-45b9-8484-c7a331b5ac03-tmp-dir\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:24.513099 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.512965 4440 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f72d419-0877-4e49-9a85-095ddea8c790-dshm\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:24.549269 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.549228 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:24.615551 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.614137 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b26d9f70-bea6-42e0-b5d2-c1b394501ddf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b26d9f70-bea6-42e0-b5d2-c1b394501ddf-maas-api-openshift-ca\") pod \"b26d9f70-bea6-42e0-b5d2-c1b394501ddf\" (UID: \"b26d9f70-bea6-42e0-b5d2-c1b394501ddf\") " Oct 31 01:14:24.615551 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.614256 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b26d9f70-bea6-42e0-b5d2-c1b394501ddf-kube-api-access-w8x5k\" (UniqueName: \"kubernetes.io/projected/b26d9f70-bea6-42e0-b5d2-c1b394501ddf-kube-api-access-w8x5k\") pod \"b26d9f70-bea6-42e0-b5d2-c1b394501ddf\" (UID: \"b26d9f70-bea6-42e0-b5d2-c1b394501ddf\") " Oct 31 01:14:24.615551 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.614312 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b26d9f70-bea6-42e0-b5d2-c1b394501ddf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b26d9f70-bea6-42e0-b5d2-c1b394501ddf-maas-api-xks-ca\") pod \"b26d9f70-bea6-42e0-b5d2-c1b394501ddf\" (UID: \"b26d9f70-bea6-42e0-b5d2-c1b394501ddf\") " Oct 31 01:14:24.615551 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.614813 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b26d9f70-bea6-42e0-b5d2-c1b394501ddf-maas-api-openshift-ca" pod "b26d9f70-bea6-42e0-b5d2-c1b394501ddf" (UID: "b26d9f70-bea6-42e0-b5d2-c1b394501ddf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:24.616404 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.616364 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b26d9f70-bea6-42e0-b5d2-c1b394501ddf-maas-api-openshift-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:24.617595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.617559 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b26d9f70-bea6-42e0-b5d2-c1b394501ddf-maas-api-xks-ca" pod "b26d9f70-bea6-42e0-b5d2-c1b394501ddf" (UID: "b26d9f70-bea6-42e0-b5d2-c1b394501ddf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:14:24.617767 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.617734 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b26d9f70-bea6-42e0-b5d2-c1b394501ddf-kube-api-access-w8x5k" pod "b26d9f70-bea6-42e0-b5d2-c1b394501ddf" (UID: "b26d9f70-bea6-42e0-b5d2-c1b394501ddf"). InnerVolumeSpecName "kube-api-access-w8x5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:24.719338 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.719300 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8x5k\" (UniqueName: \"kubernetes.io/projected/b26d9f70-bea6-42e0-b5d2-c1b394501ddf-kube-api-access-w8x5k\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:24.719338 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:24.719340 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b26d9f70-bea6-42e0-b5d2-c1b394501ddf-maas-api-xks-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:25.109960 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.109914 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" Oct 31 01:14:25.110842 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.109919 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" event={"ID":"0b82b50e-bfa7-45b9-8484-c7a331b5ac03","Type":"ContainerDied","Data":"b4882a9be9279714df783f4223f13f44a2c11556ef05b488c5731dc81346d4ab"} Oct 31 01:14:25.110842 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.110062 4440 scope.go:122] "RemoveContainer" containerID="d7b84ec0213b0cea20e0bb3b852fa1cecec5bbda6f4d8501bc5735874050bed8" Oct 31 01:14:25.111944 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.111923 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-28ff08-82b9g" Oct 31 01:14:25.112064 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.112035 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-28ff08-82b9g" event={"ID":"b26d9f70-bea6-42e0-b5d2-c1b394501ddf","Type":"ContainerDied","Data":"d16869d443d751783c1716477f019ba8901515638e9a0ca320ca7e0c8c2c65d4"} Oct 31 01:14:25.112142 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.112127 4440 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d16869d443d751783c1716477f019ba8901515638e9a0ca320ca7e0c8c2c65d4" Oct 31 01:14:25.114016 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.113994 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" Oct 31 01:14:25.114237 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.114030 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" event={"ID":"2f72d419-0877-4e49-9a85-095ddea8c790","Type":"ContainerDied","Data":"76eb3fb62f49e71ec6002a9edfa1219d0ea07306c08c9f9742ef5a16f72bd9f5"} Oct 31 01:14:25.126659 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.126635 4440 scope.go:122] "RemoveContainer" containerID="2379083646d1c3ced10003652094f928ee54c73768e44bb54a990a7dd059da98" Oct 31 01:14:25.146199 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.145981 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w"] Oct 31 01:14:25.151449 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.151406 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w"] Oct 31 01:14:25.173310 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.173266 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx"] Oct 31 01:14:25.185644 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.185608 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx"] Oct 31 01:14:25.203357 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.203301 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-2-w4-8499c8-kserve-778bf48555-wpxwx" podUID="2f72d419-0877-4e49-9a85-095ddea8c790" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:25.290638 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.290581 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8499c8/distinct-w4-8499c8-kserve-85599677b8-8tb2w" podUID="0b82b50e-bfa7-45b9-8484-c7a331b5ac03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:14:25.304210 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.304156 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:25.310663 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.310618 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878"] Oct 31 01:14:25.316578 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.316538 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw"] Oct 31 01:14:25.420731 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.420681 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:25.558254 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.558080 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:25.781092 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.781055 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b82b50e-bfa7-45b9-8484-c7a331b5ac03" path="/var/lib/kubelet/pods/0b82b50e-bfa7-45b9-8484-c7a331b5ac03/volumes" Oct 31 01:14:25.781809 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.781782 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f72d419-0877-4e49-9a85-095ddea8c790" path="/var/lib/kubelet/pods/2f72d419-0877-4e49-9a85-095ddea8c790/volumes" Oct 31 01:14:25.789541 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.789387 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" Oct 31 01:14:25.874055 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.874024 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" Oct 31 01:14:25.942541 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.937615 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" Oct 31 01:14:25.954528 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.954486 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/961cf83c-7198-4f01-9a04-00d48bb61406-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/961cf83c-7198-4f01-9a04-00d48bb61406-credential-socket\") pod \"961cf83c-7198-4f01-9a04-00d48bb61406\" (UID: \"961cf83c-7198-4f01-9a04-00d48bb61406\") " Oct 31 01:14:25.954698 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.954572 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eb48867-402c-40ca-8db3-c513636cbf20-istio-token\" (UniqueName: \"kubernetes.io/projected/4eb48867-402c-40ca-8db3-c513636cbf20-istio-token\") pod \"4eb48867-402c-40ca-8db3-c513636cbf20\" (UID: \"4eb48867-402c-40ca-8db3-c513636cbf20\") " Oct 31 01:14:25.954781 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.954698 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/961cf83c-7198-4f01-9a04-00d48bb61406-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/961cf83c-7198-4f01-9a04-00d48bb61406-workload-certs\") pod \"961cf83c-7198-4f01-9a04-00d48bb61406\" (UID: \"961cf83c-7198-4f01-9a04-00d48bb61406\") " Oct 31 01:14:25.954781 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.954739 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eb48867-402c-40ca-8db3-c513636cbf20-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eb48867-402c-40ca-8db3-c513636cbf20-istiod-ca-cert\") pod \"4eb48867-402c-40ca-8db3-c513636cbf20\" (UID: \"4eb48867-402c-40ca-8db3-c513636cbf20\") " Oct 31 01:14:25.954781 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.954776 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/961cf83c-7198-4f01-9a04-00d48bb61406-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/961cf83c-7198-4f01-9a04-00d48bb61406-istiod-ca-cert\") pod \"961cf83c-7198-4f01-9a04-00d48bb61406\" (UID: \"961cf83c-7198-4f01-9a04-00d48bb61406\") " Oct 31 01:14:25.954955 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.954810 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eb48867-402c-40ca-8db3-c513636cbf20-kube-api-access-jzm4z\" (UniqueName: \"kubernetes.io/projected/4eb48867-402c-40ca-8db3-c513636cbf20-kube-api-access-jzm4z\") pod \"4eb48867-402c-40ca-8db3-c513636cbf20\" (UID: \"4eb48867-402c-40ca-8db3-c513636cbf20\") " Oct 31 01:14:25.954955 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.954852 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/961cf83c-7198-4f01-9a04-00d48bb61406-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/961cf83c-7198-4f01-9a04-00d48bb61406-workload-socket\") pod \"961cf83c-7198-4f01-9a04-00d48bb61406\" (UID: \"961cf83c-7198-4f01-9a04-00d48bb61406\") " Oct 31 01:14:25.954955 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.954914 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/961cf83c-7198-4f01-9a04-00d48bb61406-istio-token\" (UniqueName: \"kubernetes.io/projected/961cf83c-7198-4f01-9a04-00d48bb61406-istio-token\") pod \"961cf83c-7198-4f01-9a04-00d48bb61406\" (UID: \"961cf83c-7198-4f01-9a04-00d48bb61406\") " Oct 31 01:14:25.954955 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.954805 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/961cf83c-7198-4f01-9a04-00d48bb61406-credential-socket" pod "961cf83c-7198-4f01-9a04-00d48bb61406" (UID: "961cf83c-7198-4f01-9a04-00d48bb61406"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:25.954955 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.954947 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb48867-402c-40ca-8db3-c513636cbf20-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb48867-402c-40ca-8db3-c513636cbf20-credential-socket\") pod \"4eb48867-402c-40ca-8db3-c513636cbf20\" (UID: \"4eb48867-402c-40ca-8db3-c513636cbf20\") " Oct 31 01:14:25.955194 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.954993 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/961cf83c-7198-4f01-9a04-00d48bb61406-workload-certs" pod "961cf83c-7198-4f01-9a04-00d48bb61406" (UID: "961cf83c-7198-4f01-9a04-00d48bb61406"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:25.955194 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.955046 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb48867-402c-40ca-8db3-c513636cbf20-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eb48867-402c-40ca-8db3-c513636cbf20-istio-envoy\") pod \"4eb48867-402c-40ca-8db3-c513636cbf20\" (UID: \"4eb48867-402c-40ca-8db3-c513636cbf20\") " Oct 31 01:14:25.955194 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.955117 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb48867-402c-40ca-8db3-c513636cbf20-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eb48867-402c-40ca-8db3-c513636cbf20-istio-data\") pod \"4eb48867-402c-40ca-8db3-c513636cbf20\" (UID: \"4eb48867-402c-40ca-8db3-c513636cbf20\") " Oct 31 01:14:25.955194 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.955178 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/961cf83c-7198-4f01-9a04-00d48bb61406-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/961cf83c-7198-4f01-9a04-00d48bb61406-istio-podinfo\") pod \"961cf83c-7198-4f01-9a04-00d48bb61406\" (UID: \"961cf83c-7198-4f01-9a04-00d48bb61406\") " Oct 31 01:14:25.955385 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.955227 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/961cf83c-7198-4f01-9a04-00d48bb61406-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/961cf83c-7198-4f01-9a04-00d48bb61406-istio-envoy\") pod \"961cf83c-7198-4f01-9a04-00d48bb61406\" (UID: \"961cf83c-7198-4f01-9a04-00d48bb61406\") " Oct 31 01:14:25.955385 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.955255 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/961cf83c-7198-4f01-9a04-00d48bb61406-kube-api-access-tpp7v\" (UniqueName: \"kubernetes.io/projected/961cf83c-7198-4f01-9a04-00d48bb61406-kube-api-access-tpp7v\") pod \"961cf83c-7198-4f01-9a04-00d48bb61406\" (UID: \"961cf83c-7198-4f01-9a04-00d48bb61406\") " Oct 31 01:14:25.955385 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.955272 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/961cf83c-7198-4f01-9a04-00d48bb61406-istiod-ca-cert" pod "961cf83c-7198-4f01-9a04-00d48bb61406" (UID: "961cf83c-7198-4f01-9a04-00d48bb61406"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:25.955385 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.955352 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4eb48867-402c-40ca-8db3-c513636cbf20-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eb48867-402c-40ca-8db3-c513636cbf20-istio-podinfo\") pod \"4eb48867-402c-40ca-8db3-c513636cbf20\" (UID: \"4eb48867-402c-40ca-8db3-c513636cbf20\") " Oct 31 01:14:25.955603 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.955386 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb48867-402c-40ca-8db3-c513636cbf20-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb48867-402c-40ca-8db3-c513636cbf20-workload-certs\") pod \"4eb48867-402c-40ca-8db3-c513636cbf20\" (UID: \"4eb48867-402c-40ca-8db3-c513636cbf20\") " Oct 31 01:14:25.955603 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.955498 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb48867-402c-40ca-8db3-c513636cbf20-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb48867-402c-40ca-8db3-c513636cbf20-workload-socket\") pod \"4eb48867-402c-40ca-8db3-c513636cbf20\" (UID: \"4eb48867-402c-40ca-8db3-c513636cbf20\") " Oct 31 01:14:25.955603 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.955529 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/961cf83c-7198-4f01-9a04-00d48bb61406-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/961cf83c-7198-4f01-9a04-00d48bb61406-istio-data\") pod \"961cf83c-7198-4f01-9a04-00d48bb61406\" (UID: \"961cf83c-7198-4f01-9a04-00d48bb61406\") " Oct 31 01:14:25.956058 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.955389 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eb48867-402c-40ca-8db3-c513636cbf20-istiod-ca-cert" pod "4eb48867-402c-40ca-8db3-c513636cbf20" (UID: "4eb48867-402c-40ca-8db3-c513636cbf20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:25.956058 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.955487 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb48867-402c-40ca-8db3-c513636cbf20-istio-data" pod "4eb48867-402c-40ca-8db3-c513636cbf20" (UID: "4eb48867-402c-40ca-8db3-c513636cbf20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:25.956058 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.955618 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb48867-402c-40ca-8db3-c513636cbf20-credential-socket" pod "4eb48867-402c-40ca-8db3-c513636cbf20" (UID: "4eb48867-402c-40ca-8db3-c513636cbf20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:25.956058 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.955661 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/961cf83c-7198-4f01-9a04-00d48bb61406-workload-socket" pod "961cf83c-7198-4f01-9a04-00d48bb61406" (UID: "961cf83c-7198-4f01-9a04-00d48bb61406"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:25.956373 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.956299 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb48867-402c-40ca-8db3-c513636cbf20-workload-socket" pod "4eb48867-402c-40ca-8db3-c513636cbf20" (UID: "4eb48867-402c-40ca-8db3-c513636cbf20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:25.956519 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.956491 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb48867-402c-40ca-8db3-c513636cbf20-workload-certs" pod "4eb48867-402c-40ca-8db3-c513636cbf20" (UID: "4eb48867-402c-40ca-8db3-c513636cbf20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:25.956740 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.956718 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/961cf83c-7198-4f01-9a04-00d48bb61406-istio-data" pod "961cf83c-7198-4f01-9a04-00d48bb61406" (UID: "961cf83c-7198-4f01-9a04-00d48bb61406"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:25.958143 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.958106 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb48867-402c-40ca-8db3-c513636cbf20-istio-token" pod "4eb48867-402c-40ca-8db3-c513636cbf20" (UID: "4eb48867-402c-40ca-8db3-c513636cbf20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:25.958658 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.958632 4440 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb48867-402c-40ca-8db3-c513636cbf20-workload-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:25.958761 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.958661 4440 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb48867-402c-40ca-8db3-c513636cbf20-workload-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:25.958761 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.958676 4440 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/961cf83c-7198-4f01-9a04-00d48bb61406-istio-data\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:25.958761 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.958688 4440 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/961cf83c-7198-4f01-9a04-00d48bb61406-credential-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:25.958761 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.958701 4440 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eb48867-402c-40ca-8db3-c513636cbf20-istio-token\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:25.958761 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.958713 4440 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/961cf83c-7198-4f01-9a04-00d48bb61406-workload-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:25.958761 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.958724 4440 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eb48867-402c-40ca-8db3-c513636cbf20-istiod-ca-cert\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:25.958761 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.958735 4440 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/961cf83c-7198-4f01-9a04-00d48bb61406-istiod-ca-cert\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:25.958761 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.958747 4440 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/961cf83c-7198-4f01-9a04-00d48bb61406-workload-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:25.958761 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.958759 4440 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb48867-402c-40ca-8db3-c513636cbf20-credential-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:25.959247 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.958772 4440 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eb48867-402c-40ca-8db3-c513636cbf20-istio-data\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:25.961536 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.961512 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4eb48867-402c-40ca-8db3-c513636cbf20-istio-podinfo" pod "4eb48867-402c-40ca-8db3-c513636cbf20" (UID: "4eb48867-402c-40ca-8db3-c513636cbf20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:14:25.962075 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.962038 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/961cf83c-7198-4f01-9a04-00d48bb61406-istio-envoy" pod "961cf83c-7198-4f01-9a04-00d48bb61406" (UID: "961cf83c-7198-4f01-9a04-00d48bb61406"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:25.974706 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.973884 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/961cf83c-7198-4f01-9a04-00d48bb61406-istio-podinfo" pod "961cf83c-7198-4f01-9a04-00d48bb61406" (UID: "961cf83c-7198-4f01-9a04-00d48bb61406"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:14:25.974706 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.974197 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb48867-402c-40ca-8db3-c513636cbf20-istio-envoy" pod "4eb48867-402c-40ca-8db3-c513636cbf20" (UID: "4eb48867-402c-40ca-8db3-c513636cbf20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:25.974706 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.974587 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/961cf83c-7198-4f01-9a04-00d48bb61406-kube-api-access-tpp7v" pod "961cf83c-7198-4f01-9a04-00d48bb61406" (UID: "961cf83c-7198-4f01-9a04-00d48bb61406"). InnerVolumeSpecName "kube-api-access-tpp7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:25.974706 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.974629 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/961cf83c-7198-4f01-9a04-00d48bb61406-istio-token" pod "961cf83c-7198-4f01-9a04-00d48bb61406" (UID: "961cf83c-7198-4f01-9a04-00d48bb61406"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:25.976293 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.976252 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb48867-402c-40ca-8db3-c513636cbf20-kube-api-access-jzm4z" pod "4eb48867-402c-40ca-8db3-c513636cbf20" (UID: "4eb48867-402c-40ca-8db3-c513636cbf20"). InnerVolumeSpecName "kube-api-access-jzm4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:25.992272 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:25.992183 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:26.059407 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.059365 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a46413-c078-447a-9bbb-d9ea9431469e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a46413-c078-447a-9bbb-d9ea9431469e-workload-socket\") pod \"81a46413-c078-447a-9bbb-d9ea9431469e\" (UID: \"81a46413-c078-447a-9bbb-d9ea9431469e\") " Oct 31 01:14:26.059629 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.059478 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81a46413-c078-447a-9bbb-d9ea9431469e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a46413-c078-447a-9bbb-d9ea9431469e-istiod-ca-cert\") pod \"81a46413-c078-447a-9bbb-d9ea9431469e\" (UID: \"81a46413-c078-447a-9bbb-d9ea9431469e\") " Oct 31 01:14:26.059629 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.059497 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a46413-c078-447a-9bbb-d9ea9431469e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a46413-c078-447a-9bbb-d9ea9431469e-credential-socket\") pod \"81a46413-c078-447a-9bbb-d9ea9431469e\" (UID: \"81a46413-c078-447a-9bbb-d9ea9431469e\") " Oct 31 01:14:26.059629 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.059528 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81a46413-c078-447a-9bbb-d9ea9431469e-kube-api-access-hrg6l\" (UniqueName: \"kubernetes.io/projected/81a46413-c078-447a-9bbb-d9ea9431469e-kube-api-access-hrg6l\") pod \"81a46413-c078-447a-9bbb-d9ea9431469e\" (UID: \"81a46413-c078-447a-9bbb-d9ea9431469e\") " Oct 31 01:14:26.059629 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.059554 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81a46413-c078-447a-9bbb-d9ea9431469e-istio-token\" (UniqueName: \"kubernetes.io/projected/81a46413-c078-447a-9bbb-d9ea9431469e-istio-token\") pod \"81a46413-c078-447a-9bbb-d9ea9431469e\" (UID: \"81a46413-c078-447a-9bbb-d9ea9431469e\") " Oct 31 01:14:26.059831 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.059634 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81a46413-c078-447a-9bbb-d9ea9431469e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a46413-c078-447a-9bbb-d9ea9431469e-istio-podinfo\") pod \"81a46413-c078-447a-9bbb-d9ea9431469e\" (UID: \"81a46413-c078-447a-9bbb-d9ea9431469e\") " Oct 31 01:14:26.059831 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.059680 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a46413-c078-447a-9bbb-d9ea9431469e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a46413-c078-447a-9bbb-d9ea9431469e-istio-envoy\") pod \"81a46413-c078-447a-9bbb-d9ea9431469e\" (UID: \"81a46413-c078-447a-9bbb-d9ea9431469e\") " Oct 31 01:14:26.059831 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.059697 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a46413-c078-447a-9bbb-d9ea9431469e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a46413-c078-447a-9bbb-d9ea9431469e-workload-certs\") pod \"81a46413-c078-447a-9bbb-d9ea9431469e\" (UID: \"81a46413-c078-447a-9bbb-d9ea9431469e\") " Oct 31 01:14:26.059831 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.059759 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a46413-c078-447a-9bbb-d9ea9431469e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a46413-c078-447a-9bbb-d9ea9431469e-istio-data\") pod \"81a46413-c078-447a-9bbb-d9ea9431469e\" (UID: \"81a46413-c078-447a-9bbb-d9ea9431469e\") " Oct 31 01:14:26.060022 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.059952 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a46413-c078-447a-9bbb-d9ea9431469e-credential-socket" pod "81a46413-c078-447a-9bbb-d9ea9431469e" (UID: "81a46413-c078-447a-9bbb-d9ea9431469e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:26.060368 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.060337 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a46413-c078-447a-9bbb-d9ea9431469e-workload-socket" pod "81a46413-c078-447a-9bbb-d9ea9431469e" (UID: "81a46413-c078-447a-9bbb-d9ea9431469e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:26.060687 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.060592 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzm4z\" (UniqueName: \"kubernetes.io/projected/4eb48867-402c-40ca-8db3-c513636cbf20-kube-api-access-jzm4z\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:26.060687 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.060624 4440 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/961cf83c-7198-4f01-9a04-00d48bb61406-istio-token\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:26.060687 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.060642 4440 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eb48867-402c-40ca-8db3-c513636cbf20-istio-envoy\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:26.060687 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.060656 4440 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/961cf83c-7198-4f01-9a04-00d48bb61406-istio-podinfo\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:26.060687 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.060661 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81a46413-c078-447a-9bbb-d9ea9431469e-istiod-ca-cert" pod "81a46413-c078-447a-9bbb-d9ea9431469e" (UID: "81a46413-c078-447a-9bbb-d9ea9431469e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:26.060687 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.060670 4440 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a46413-c078-447a-9bbb-d9ea9431469e-workload-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:26.061094 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.060715 4440 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/961cf83c-7198-4f01-9a04-00d48bb61406-istio-envoy\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:26.061094 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.060738 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpp7v\" (UniqueName: \"kubernetes.io/projected/961cf83c-7198-4f01-9a04-00d48bb61406-kube-api-access-tpp7v\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:26.061094 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.060757 4440 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eb48867-402c-40ca-8db3-c513636cbf20-istio-podinfo\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:26.061094 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.060774 4440 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a46413-c078-447a-9bbb-d9ea9431469e-credential-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:26.061094 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.060966 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a46413-c078-447a-9bbb-d9ea9431469e-workload-certs" pod "81a46413-c078-447a-9bbb-d9ea9431469e" (UID: "81a46413-c078-447a-9bbb-d9ea9431469e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:26.061094 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.060986 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a46413-c078-447a-9bbb-d9ea9431469e-istio-data" pod "81a46413-c078-447a-9bbb-d9ea9431469e" (UID: "81a46413-c078-447a-9bbb-d9ea9431469e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:26.063061 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.063037 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81a46413-c078-447a-9bbb-d9ea9431469e-istio-podinfo" pod "81a46413-c078-447a-9bbb-d9ea9431469e" (UID: "81a46413-c078-447a-9bbb-d9ea9431469e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:14:26.063135 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.063100 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81a46413-c078-447a-9bbb-d9ea9431469e-kube-api-access-hrg6l" pod "81a46413-c078-447a-9bbb-d9ea9431469e" (UID: "81a46413-c078-447a-9bbb-d9ea9431469e"). InnerVolumeSpecName "kube-api-access-hrg6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:26.063241 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.063222 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a46413-c078-447a-9bbb-d9ea9431469e-istio-envoy" pod "81a46413-c078-447a-9bbb-d9ea9431469e" (UID: "81a46413-c078-447a-9bbb-d9ea9431469e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:26.063290 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.063273 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81a46413-c078-447a-9bbb-d9ea9431469e-istio-token" pod "81a46413-c078-447a-9bbb-d9ea9431469e" (UID: "81a46413-c078-447a-9bbb-d9ea9431469e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:26.127394 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.127353 4440 generic.go:356] "Generic (PLEG): container finished" podID="81a46413-c078-447a-9bbb-d9ea9431469e" containerID="3b795f03e8a4813ba1a35e62abc282d37d77cc108e93b21613fd53c0d1dc8e41" exitCode=0 Oct 31 01:14:26.127593 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.127476 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" Oct 31 01:14:26.127593 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.127539 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" event={"ID":"81a46413-c078-447a-9bbb-d9ea9431469e","Type":"ContainerDied","Data":"3b795f03e8a4813ba1a35e62abc282d37d77cc108e93b21613fd53c0d1dc8e41"} Oct 31 01:14:26.127696 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.127610 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d" event={"ID":"81a46413-c078-447a-9bbb-d9ea9431469e","Type":"ContainerDied","Data":"b736ec712d17344a6e4203dbcfaba1845c5679bb699572c1a52b0f4b6c9036d9"} Oct 31 01:14:26.127696 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.127641 4440 scope.go:122] "RemoveContainer" containerID="3b795f03e8a4813ba1a35e62abc282d37d77cc108e93b21613fd53c0d1dc8e41" Oct 31 01:14:26.138460 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.131981 4440 generic.go:356] "Generic (PLEG): container finished" podID="961cf83c-7198-4f01-9a04-00d48bb61406" containerID="49ee265d5d7615a3f2f6d41e7ad9c12dbe984d1ff9a1b1cf834858150420f6b6" exitCode=0 Oct 31 01:14:26.138460 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.132024 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" Oct 31 01:14:26.138460 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.132072 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" event={"ID":"961cf83c-7198-4f01-9a04-00d48bb61406","Type":"ContainerDied","Data":"49ee265d5d7615a3f2f6d41e7ad9c12dbe984d1ff9a1b1cf834858150420f6b6"} Oct 31 01:14:26.138460 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.132116 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx" event={"ID":"961cf83c-7198-4f01-9a04-00d48bb61406","Type":"ContainerDied","Data":"2842ff4b211268b472602e0a1ed7cf7bdc57fc41b58cc81c63c7ea9669faee39"} Oct 31 01:14:26.144600 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.144575 4440 scope.go:122] "RemoveContainer" containerID="cb4b0e121cee63b353d20ad49269a0abf7758c19fdb73b48fd1d8b452a7a720d" Oct 31 01:14:26.144805 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.144721 4440 generic.go:356] "Generic (PLEG): container finished" podID="4eb48867-402c-40ca-8db3-c513636cbf20" containerID="524b582fa2d49e50204960dff4f1821661b4b193b9f2f4aec0a23d8183a3764b" exitCode=0 Oct 31 01:14:26.144805 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.144770 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" event={"ID":"4eb48867-402c-40ca-8db3-c513636cbf20","Type":"ContainerDied","Data":"524b582fa2d49e50204960dff4f1821661b4b193b9f2f4aec0a23d8183a3764b"} Oct 31 01:14:26.144805 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.144789 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" Oct 31 01:14:26.144981 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.144821 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq" event={"ID":"4eb48867-402c-40ca-8db3-c513636cbf20","Type":"ContainerDied","Data":"5389c5e1689ac82c5762c8f01eff6169a7b1e6cc64a7432df01059bff28f8669"} Oct 31 01:14:26.169520 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.163568 4440 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a46413-c078-447a-9bbb-d9ea9431469e-istiod-ca-cert\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:26.169520 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.163600 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrg6l\" (UniqueName: \"kubernetes.io/projected/81a46413-c078-447a-9bbb-d9ea9431469e-kube-api-access-hrg6l\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:26.169520 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.163625 4440 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81a46413-c078-447a-9bbb-d9ea9431469e-istio-token\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:26.169520 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.163641 4440 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a46413-c078-447a-9bbb-d9ea9431469e-istio-podinfo\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:26.169520 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.163656 4440 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a46413-c078-447a-9bbb-d9ea9431469e-istio-envoy\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:26.169520 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.163673 4440 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a46413-c078-447a-9bbb-d9ea9431469e-workload-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:26.169520 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.163694 4440 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a46413-c078-447a-9bbb-d9ea9431469e-istio-data\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:26.169520 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.169446 4440 scope.go:122] "RemoveContainer" containerID="3b795f03e8a4813ba1a35e62abc282d37d77cc108e93b21613fd53c0d1dc8e41" Oct 31 01:14:26.169960 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.169609 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d"] Oct 31 01:14:26.171200 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:14:26.170795 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b795f03e8a4813ba1a35e62abc282d37d77cc108e93b21613fd53c0d1dc8e41\": container with ID starting with 3b795f03e8a4813ba1a35e62abc282d37d77cc108e93b21613fd53c0d1dc8e41 not found: ID does not exist" containerID="3b795f03e8a4813ba1a35e62abc282d37d77cc108e93b21613fd53c0d1dc8e41" Oct 31 01:14:26.171200 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.170840 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b795f03e8a4813ba1a35e62abc282d37d77cc108e93b21613fd53c0d1dc8e41"} err="failed to get container status \"3b795f03e8a4813ba1a35e62abc282d37d77cc108e93b21613fd53c0d1dc8e41\": rpc error: code = NotFound desc = could not find container \"3b795f03e8a4813ba1a35e62abc282d37d77cc108e93b21613fd53c0d1dc8e41\": container with ID starting with 3b795f03e8a4813ba1a35e62abc282d37d77cc108e93b21613fd53c0d1dc8e41 not found: ID does not exist" Oct 31 01:14:26.171200 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.170865 4440 scope.go:122] "RemoveContainer" containerID="cb4b0e121cee63b353d20ad49269a0abf7758c19fdb73b48fd1d8b452a7a720d" Oct 31 01:14:26.171496 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:14:26.171305 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb4b0e121cee63b353d20ad49269a0abf7758c19fdb73b48fd1d8b452a7a720d\": container with ID starting with cb4b0e121cee63b353d20ad49269a0abf7758c19fdb73b48fd1d8b452a7a720d not found: ID does not exist" containerID="cb4b0e121cee63b353d20ad49269a0abf7758c19fdb73b48fd1d8b452a7a720d" Oct 31 01:14:26.171496 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.171348 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb4b0e121cee63b353d20ad49269a0abf7758c19fdb73b48fd1d8b452a7a720d"} err="failed to get container status \"cb4b0e121cee63b353d20ad49269a0abf7758c19fdb73b48fd1d8b452a7a720d\": rpc error: code = NotFound desc = could not find container \"cb4b0e121cee63b353d20ad49269a0abf7758c19fdb73b48fd1d8b452a7a720d\": container with ID starting with cb4b0e121cee63b353d20ad49269a0abf7758c19fdb73b48fd1d8b452a7a720d not found: ID does not exist" Oct 31 01:14:26.171496 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.171370 4440 scope.go:122] "RemoveContainer" containerID="49ee265d5d7615a3f2f6d41e7ad9c12dbe984d1ff9a1b1cf834858150420f6b6" Oct 31 01:14:26.177942 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.177910 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-7gz7d"] Oct 31 01:14:26.184673 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.184641 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx"] Oct 31 01:14:26.186660 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.186638 4440 scope.go:122] "RemoveContainer" containerID="49ee265d5d7615a3f2f6d41e7ad9c12dbe984d1ff9a1b1cf834858150420f6b6" Oct 31 01:14:26.187005 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:14:26.186984 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49ee265d5d7615a3f2f6d41e7ad9c12dbe984d1ff9a1b1cf834858150420f6b6\": container with ID starting with 49ee265d5d7615a3f2f6d41e7ad9c12dbe984d1ff9a1b1cf834858150420f6b6 not found: ID does not exist" containerID="49ee265d5d7615a3f2f6d41e7ad9c12dbe984d1ff9a1b1cf834858150420f6b6" Oct 31 01:14:26.187066 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.187022 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49ee265d5d7615a3f2f6d41e7ad9c12dbe984d1ff9a1b1cf834858150420f6b6"} err="failed to get container status \"49ee265d5d7615a3f2f6d41e7ad9c12dbe984d1ff9a1b1cf834858150420f6b6\": rpc error: code = NotFound desc = could not find container \"49ee265d5d7615a3f2f6d41e7ad9c12dbe984d1ff9a1b1cf834858150420f6b6\": container with ID starting with 49ee265d5d7615a3f2f6d41e7ad9c12dbe984d1ff9a1b1cf834858150420f6b6 not found: ID does not exist" Oct 31 01:14:26.187066 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.187041 4440 scope.go:122] "RemoveContainer" containerID="524b582fa2d49e50204960dff4f1821661b4b193b9f2f4aec0a23d8183a3764b" Oct 31 01:14:26.189638 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.189613 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-zn9sx"] Oct 31 01:14:26.194756 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.194696 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq"] Oct 31 01:14:26.199531 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.199493 4440 scope.go:122] "RemoveContainer" containerID="524b582fa2d49e50204960dff4f1821661b4b193b9f2f4aec0a23d8183a3764b" Oct 31 01:14:26.199880 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:14:26.199858 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"524b582fa2d49e50204960dff4f1821661b4b193b9f2f4aec0a23d8183a3764b\": container with ID starting with 524b582fa2d49e50204960dff4f1821661b4b193b9f2f4aec0a23d8183a3764b not found: ID does not exist" containerID="524b582fa2d49e50204960dff4f1821661b4b193b9f2f4aec0a23d8183a3764b" Oct 31 01:14:26.199961 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.199889 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"524b582fa2d49e50204960dff4f1821661b4b193b9f2f4aec0a23d8183a3764b"} err="failed to get container status \"524b582fa2d49e50204960dff4f1821661b4b193b9f2f4aec0a23d8183a3764b\": rpc error: code = NotFound desc = could not find container \"524b582fa2d49e50204960dff4f1821661b4b193b9f2f4aec0a23d8183a3764b\": container with ID starting with 524b582fa2d49e50204960dff4f1821661b4b193b9f2f4aec0a23d8183a3764b not found: ID does not exist" Oct 31 01:14:26.201102 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.201074 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-1b8d05-openshift-default-659bf8c59c-jwnlq"] Oct 31 01:14:26.797039 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:26.796988 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:27.570120 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:27.570086 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:27.772850 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:27.772813 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eb48867-402c-40ca-8db3-c513636cbf20" path="/var/lib/kubelet/pods/4eb48867-402c-40ca-8db3-c513636cbf20/volumes" Oct 31 01:14:27.773457 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:27.773411 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81a46413-c078-447a-9bbb-d9ea9431469e" path="/var/lib/kubelet/pods/81a46413-c078-447a-9bbb-d9ea9431469e/volumes" Oct 31 01:14:27.773891 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:27.773877 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="961cf83c-7198-4f01-9a04-00d48bb61406" path="/var/lib/kubelet/pods/961cf83c-7198-4f01-9a04-00d48bb61406/volumes" Oct 31 01:14:28.289965 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:28.289926 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:29.189855 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:29.189813 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:29.377489 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:29.377455 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:29.478271 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:29.478172 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8499c8-5c65ddd74c-9r5n8" podUID="ab8ba016-584a-4646-a3d4-f6db4ccf0700" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:14:29.495933 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:29.495892 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:29.697986 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:29.697944 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts"] Oct 31 01:14:29.698259 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:29.698227 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" podUID="583a1183-7c84-40ce-aecb-9663f82a8a47" containerName="istio-proxy" containerID="cri-o://b6de0db241e1dcd09d56cd8b6d36c57f1a8e0a80d13e136c1b7080cc5d4f5044" gracePeriod=30 Oct 31 01:14:29.705137 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:29.704969 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6"] Oct 31 01:14:29.705386 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:29.705348 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" podUID="dc34b9f5-85b1-4cab-a49e-c2106d685cc2" containerName="istio-proxy" containerID="cri-o://107e6b0b6110a5e9993d7259cfe27fa51563593824c65fc1446ac3ea96c881b7" gracePeriod=30 Oct 31 01:14:29.724844 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:29.722035 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq"] Oct 31 01:14:29.724844 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:29.722387 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" podUID="eeb7394c-0c4a-43e6-bbca-f4222bec42ff" containerName="istio-proxy" containerID="cri-o://ce2378e90758e17c8ae084350a07551e3ccb018ee1faddd45d9b880089b8a937" gracePeriod=30 Oct 31 01:14:29.731589 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:29.730332 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5"] Oct 31 01:14:29.732771 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:29.732659 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" podUID="f889944d-e872-4b84-baa6-829de12f3ba1" containerName="istio-proxy" containerID="cri-o://7e9c4a7ae854dee3a08c0a09f9e8767db889c6b01e7efb79faa5c8464e615674" gracePeriod=30 Oct 31 01:14:29.763124 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:29.763077 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:30.151291 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:30.151199 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:14:30.151511 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:30.151281 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" podUID="dc34b9f5-85b1-4cab-a49e-c2106d685cc2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:14:30.332657 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:30.332617 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:30.788823 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:30.788789 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:30.904859 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:30.904823 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-28ff08-8947d86b5-frcjf"] Oct 31 01:14:30.909400 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:30.909364 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:31.019580 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:14:31.016535 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:14:31.019580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:31.018949 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:31.161453 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:31.161349 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:31.637081 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:31.637032 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:31.938704 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:31.938671 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:32.449455 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.449397 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-252485f4-f4rvq"] Oct 31 01:14:32.450744 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.450712 4440 cpu_manager.go:414] "containerMap: removing container" podUID="81a46413-c078-447a-9bbb-d9ea9431469e" containerName="istio-proxy" Oct 31 01:14:32.450744 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.450742 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81a46413-c078-447a-9bbb-d9ea9431469e" containerName="istio-proxy" Oct 31 01:14:32.450908 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.450756 4440 cpu_manager.go:414] "containerMap: removing container" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" Oct 31 01:14:32.450908 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.450763 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" Oct 31 01:14:32.450908 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.450779 4440 cpu_manager.go:414] "containerMap: removing container" podUID="0b82b50e-bfa7-45b9-8484-c7a331b5ac03" containerName="main" Oct 31 01:14:32.450908 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.450789 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b82b50e-bfa7-45b9-8484-c7a331b5ac03" containerName="main" Oct 31 01:14:32.450908 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.450808 4440 cpu_manager.go:414] "containerMap: removing container" podUID="617e420d-d029-4f4b-8ae5-63fe7d85971e" containerName="main" Oct 31 01:14:32.450908 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.450814 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="617e420d-d029-4f4b-8ae5-63fe7d85971e" containerName="main" Oct 31 01:14:32.450908 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.450837 4440 cpu_manager.go:414] "containerMap: removing container" podUID="2f72d419-0877-4e49-9a85-095ddea8c790" containerName="main" Oct 31 01:14:32.450908 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.450845 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f72d419-0877-4e49-9a85-095ddea8c790" containerName="main" Oct 31 01:14:32.450908 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.450866 4440 cpu_manager.go:414] "containerMap: removing container" podUID="81a46413-c078-447a-9bbb-d9ea9431469e" containerName="istio-proxy" Oct 31 01:14:32.450908 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.450873 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81a46413-c078-447a-9bbb-d9ea9431469e" containerName="istio-proxy" Oct 31 01:14:32.450908 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.450882 4440 cpu_manager.go:414] "containerMap: removing container" podUID="4eb48867-402c-40ca-8db3-c513636cbf20" containerName="istio-proxy" Oct 31 01:14:32.450908 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.450889 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eb48867-402c-40ca-8db3-c513636cbf20" containerName="istio-proxy" Oct 31 01:14:32.450908 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.450911 4440 cpu_manager.go:414] "containerMap: removing container" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" Oct 31 01:14:32.451445 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.450918 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" Oct 31 01:14:32.451445 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.450929 4440 cpu_manager.go:414] "containerMap: removing container" podUID="961cf83c-7198-4f01-9a04-00d48bb61406" containerName="istio-proxy" Oct 31 01:14:32.451445 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.450935 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="961cf83c-7198-4f01-9a04-00d48bb61406" containerName="istio-proxy" Oct 31 01:14:32.451445 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.450956 4440 cpu_manager.go:414] "containerMap: removing container" podUID="b26d9f70-bea6-42e0-b5d2-c1b394501ddf" containerName="revoke-keys" Oct 31 01:14:32.451445 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.450963 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b26d9f70-bea6-42e0-b5d2-c1b394501ddf" containerName="revoke-keys" Oct 31 01:14:32.451445 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.451132 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="81a46413-c078-447a-9bbb-d9ea9431469e" containerName="istio-proxy" Oct 31 01:14:32.451445 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.451148 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2ca4946-eb55-4a6b-a4bb-257a5792ffed" containerName="main" Oct 31 01:14:32.451445 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.451160 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b82b50e-bfa7-45b9-8484-c7a331b5ac03" containerName="main" Oct 31 01:14:32.451445 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.451169 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="617e420d-d029-4f4b-8ae5-63fe7d85971e" containerName="main" Oct 31 01:14:32.451445 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.451183 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f72d419-0877-4e49-9a85-095ddea8c790" containerName="main" Oct 31 01:14:32.451445 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.451195 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eb48867-402c-40ca-8db3-c513636cbf20" containerName="istio-proxy" Oct 31 01:14:32.451445 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.451207 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ab080a7-963c-45db-a83c-2c359b1bcf06" containerName="main" Oct 31 01:14:32.451445 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.451215 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="961cf83c-7198-4f01-9a04-00d48bb61406" containerName="istio-proxy" Oct 31 01:14:32.451445 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.451227 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="b26d9f70-bea6-42e0-b5d2-c1b394501ddf" containerName="revoke-keys" Oct 31 01:14:32.463450 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.458202 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-252485f4-f4rvq" Oct 31 01:14:32.463450 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.461568 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-252485f4-f4rvq"] Oct 31 01:14:32.502849 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.502816 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:14:32.503310 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.503279 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zs6l6\"" Oct 31 01:14:32.529280 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.529241 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkx55"] Oct 31 01:14:32.530125 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.530099 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="81a46413-c078-447a-9bbb-d9ea9431469e" containerName="istio-proxy" Oct 31 01:14:32.534314 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.534284 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkx55" Oct 31 01:14:32.543694 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.543663 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkx55"] Oct 31 01:14:32.602826 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.602786 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 31 01:14:32.602826 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.602818 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 31 01:14:32.603117 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.603096 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 31 01:14:32.603584 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.603567 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-clrlv\"" Oct 31 01:14:32.603953 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.603934 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkx55\" (UID: \"4f77d13d-b9ab-44cd-ba25-37f8841cf8da\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkx55" Oct 31 01:14:32.604092 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.604076 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkx55\" (UID: \"4f77d13d-b9ab-44cd-ba25-37f8841cf8da\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkx55" Oct 31 01:14:32.604134 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.604112 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvm8s\" (UniqueName: \"kubernetes.io/projected/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-kube-api-access-cvm8s\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkx55\" (UID: \"4f77d13d-b9ab-44cd-ba25-37f8841cf8da\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkx55" Oct 31 01:14:32.604187 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.604131 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkx55\" (UID: \"4f77d13d-b9ab-44cd-ba25-37f8841cf8da\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkx55" Oct 31 01:14:32.604230 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.604210 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f77962d-e14f-4225-b2e9-7a709dc8c4f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-252485f4-f4rvq\" (UID: \"7f77962d-e14f-4225-b2e9-7a709dc8c4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-252485f4-f4rvq" Oct 31 01:14:32.604283 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.604269 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkx55\" (UID: \"4f77d13d-b9ab-44cd-ba25-37f8841cf8da\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkx55" Oct 31 01:14:32.604391 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.604376 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkx55\" (UID: \"4f77d13d-b9ab-44cd-ba25-37f8841cf8da\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkx55" Oct 31 01:14:32.604542 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.604527 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsr9t\" (UniqueName: \"kubernetes.io/projected/7f77962d-e14f-4225-b2e9-7a709dc8c4f4-kube-api-access-nsr9t\") pod \"maas-api-revoke-sub-252485f4-f4rvq\" (UID: \"7f77962d-e14f-4225-b2e9-7a709dc8c4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-252485f4-f4rvq" Oct 31 01:14:32.604594 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.604562 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkx55\" (UID: \"4f77d13d-b9ab-44cd-ba25-37f8841cf8da\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkx55" Oct 31 01:14:32.604636 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.604599 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f77962d-e14f-4225-b2e9-7a709dc8c4f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-252485f4-f4rvq\" (UID: \"7f77962d-e14f-4225-b2e9-7a709dc8c4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-252485f4-f4rvq" Oct 31 01:14:32.706814 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.706710 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkx55\" (UID: \"4f77d13d-b9ab-44cd-ba25-37f8841cf8da\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkx55" Oct 31 01:14:32.706814 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.706783 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvm8s\" (UniqueName: \"kubernetes.io/projected/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-kube-api-access-cvm8s\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkx55\" (UID: \"4f77d13d-b9ab-44cd-ba25-37f8841cf8da\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkx55" Oct 31 01:14:32.706814 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.706808 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkx55\" (UID: \"4f77d13d-b9ab-44cd-ba25-37f8841cf8da\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkx55" Oct 31 01:14:32.707126 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.706864 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f77962d-e14f-4225-b2e9-7a709dc8c4f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-252485f4-f4rvq\" (UID: \"7f77962d-e14f-4225-b2e9-7a709dc8c4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-252485f4-f4rvq" Oct 31 01:14:32.707126 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.706890 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkx55\" (UID: \"4f77d13d-b9ab-44cd-ba25-37f8841cf8da\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkx55" Oct 31 01:14:32.707126 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.706939 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkx55\" (UID: \"4f77d13d-b9ab-44cd-ba25-37f8841cf8da\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkx55" Oct 31 01:14:32.707126 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.707048 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsr9t\" (UniqueName: \"kubernetes.io/projected/7f77962d-e14f-4225-b2e9-7a709dc8c4f4-kube-api-access-nsr9t\") pod \"maas-api-revoke-sub-252485f4-f4rvq\" (UID: \"7f77962d-e14f-4225-b2e9-7a709dc8c4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-252485f4-f4rvq" Oct 31 01:14:32.707126 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.707093 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkx55\" (UID: \"4f77d13d-b9ab-44cd-ba25-37f8841cf8da\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkx55" Oct 31 01:14:32.707450 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.707149 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f77962d-e14f-4225-b2e9-7a709dc8c4f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-252485f4-f4rvq\" (UID: \"7f77962d-e14f-4225-b2e9-7a709dc8c4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-252485f4-f4rvq" Oct 31 01:14:32.707450 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.707219 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkx55\" (UID: \"4f77d13d-b9ab-44cd-ba25-37f8841cf8da\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkx55" Oct 31 01:14:32.707763 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.707736 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f77962d-e14f-4225-b2e9-7a709dc8c4f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-252485f4-f4rvq\" (UID: \"7f77962d-e14f-4225-b2e9-7a709dc8c4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-252485f4-f4rvq" Oct 31 01:14:32.708002 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.707974 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkx55\" (UID: \"4f77d13d-b9ab-44cd-ba25-37f8841cf8da\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkx55" Oct 31 01:14:32.710379 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.710108 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkx55\" (UID: \"4f77d13d-b9ab-44cd-ba25-37f8841cf8da\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkx55" Oct 31 01:14:32.710379 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.710332 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkx55\" (UID: \"4f77d13d-b9ab-44cd-ba25-37f8841cf8da\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkx55" Oct 31 01:14:32.710584 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.710560 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f77962d-e14f-4225-b2e9-7a709dc8c4f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-252485f4-f4rvq\" (UID: \"7f77962d-e14f-4225-b2e9-7a709dc8c4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-252485f4-f4rvq" Oct 31 01:14:32.710643 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.710590 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkx55\" (UID: \"4f77d13d-b9ab-44cd-ba25-37f8841cf8da\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkx55" Oct 31 01:14:32.710817 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.710796 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkx55\" (UID: \"4f77d13d-b9ab-44cd-ba25-37f8841cf8da\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkx55" Oct 31 01:14:32.715776 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.715740 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvm8s\" (UniqueName: \"kubernetes.io/projected/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-kube-api-access-cvm8s\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkx55\" (UID: \"4f77d13d-b9ab-44cd-ba25-37f8841cf8da\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkx55" Oct 31 01:14:32.715952 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.715927 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsr9t\" (UniqueName: \"kubernetes.io/projected/7f77962d-e14f-4225-b2e9-7a709dc8c4f4-kube-api-access-nsr9t\") pod \"maas-api-revoke-sub-252485f4-f4rvq\" (UID: \"7f77962d-e14f-4225-b2e9-7a709dc8c4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-252485f4-f4rvq" Oct 31 01:14:32.716656 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.716631 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkx55\" (UID: \"4f77d13d-b9ab-44cd-ba25-37f8841cf8da\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkx55" Oct 31 01:14:32.776495 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.776452 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:32.783282 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.783249 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-252485f4-f4rvq" Oct 31 01:14:32.851210 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:32.850889 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkx55" Oct 31 01:14:33.116938 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:14:33.116906 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f77962d_e14f_4225_b2e9_7a709dc8c4f4.slice/crio-0e3563408a061f616a3546fcfca286a9041e5cc9434b9f36c71e35ddcbba6235 WatchSource:0}: Error finding container 0e3563408a061f616a3546fcfca286a9041e5cc9434b9f36c71e35ddcbba6235: Status 404 returned error can't find the container with id 0e3563408a061f616a3546fcfca286a9041e5cc9434b9f36c71e35ddcbba6235 Oct 31 01:14:33.117756 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:33.117721 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-252485f4-f4rvq"] Oct 31 01:14:33.175749 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:14:33.175711 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f77d13d_b9ab_44cd_ba25_37f8841cf8da.slice/crio-24c05155a0c2987e40437072a00e263ea6a87ac4a4f51fd2300cb9e774d613c8 WatchSource:0}: Error finding container 24c05155a0c2987e40437072a00e263ea6a87ac4a4f51fd2300cb9e774d613c8: Status 404 returned error can't find the container with id 24c05155a0c2987e40437072a00e263ea6a87ac4a4f51fd2300cb9e774d613c8 Oct 31 01:14:33.175938 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:33.175877 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkx55"] Oct 31 01:14:33.210280 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:33.210192 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkx55" event={"ID":"4f77d13d-b9ab-44cd-ba25-37f8841cf8da","Type":"ContainerStarted","Data":"24c05155a0c2987e40437072a00e263ea6a87ac4a4f51fd2300cb9e774d613c8"} Oct 31 01:14:33.211753 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:33.211720 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-252485f4-f4rvq" event={"ID":"7f77962d-e14f-4225-b2e9-7a709dc8c4f4","Type":"ContainerStarted","Data":"0e3563408a061f616a3546fcfca286a9041e5cc9434b9f36c71e35ddcbba6235"} Oct 31 01:14:33.586697 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:33.586581 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:34.222838 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:34.222800 4440 generic.go:356] "Generic (PLEG): container finished" podID="eaad3e7d-1293-43d5-9fe6-674e69ea840e" containerID="326d064c31cabc994f3860adf0c6c71f24c1ae6bdcd13f040b76733cd4c943c3" exitCode=0 Oct 31 01:14:34.223250 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:34.222848 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-0a6cca-6599d4945f-dlflj" event={"ID":"eaad3e7d-1293-43d5-9fe6-674e69ea840e","Type":"ContainerDied","Data":"326d064c31cabc994f3860adf0c6c71f24c1ae6bdcd13f040b76733cd4c943c3"} Oct 31 01:14:34.224999 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:34.224969 4440 generic.go:356] "Generic (PLEG): container finished" podID="7f77962d-e14f-4225-b2e9-7a709dc8c4f4" containerID="f01cc4cde3b546e2ad2e4180a9ef3d5b748b42d08b240e74228c7e99ef4a814b" exitCode=0 Oct 31 01:14:34.225132 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:34.225043 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-252485f4-f4rvq" event={"ID":"7f77962d-e14f-4225-b2e9-7a709dc8c4f4","Type":"ContainerDied","Data":"f01cc4cde3b546e2ad2e4180a9ef3d5b748b42d08b240e74228c7e99ef4a814b"} Oct 31 01:14:34.390246 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:34.390202 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:34.401703 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:34.401677 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-0a6cca-6599d4945f-dlflj" Oct 31 01:14:34.550596 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:34.550511 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaad3e7d-1293-43d5-9fe6-674e69ea840e-kube-api-access-nm6l2\" (UniqueName: \"kubernetes.io/projected/eaad3e7d-1293-43d5-9fe6-674e69ea840e-kube-api-access-nm6l2\") pod \"eaad3e7d-1293-43d5-9fe6-674e69ea840e\" (UID: \"eaad3e7d-1293-43d5-9fe6-674e69ea840e\") " Oct 31 01:14:34.550818 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:34.550801 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaad3e7d-1293-43d5-9fe6-674e69ea840e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaad3e7d-1293-43d5-9fe6-674e69ea840e-plugins-config-volume\") pod \"eaad3e7d-1293-43d5-9fe6-674e69ea840e\" (UID: \"eaad3e7d-1293-43d5-9fe6-674e69ea840e\") " Oct 31 01:14:34.551253 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:34.551223 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaad3e7d-1293-43d5-9fe6-674e69ea840e-plugins-config-volume" pod "eaad3e7d-1293-43d5-9fe6-674e69ea840e" (UID: "eaad3e7d-1293-43d5-9fe6-674e69ea840e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:34.551408 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:34.551385 4440 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaad3e7d-1293-43d5-9fe6-674e69ea840e-plugins-config-volume\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:34.552847 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:34.552819 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaad3e7d-1293-43d5-9fe6-674e69ea840e-kube-api-access-nm6l2" pod "eaad3e7d-1293-43d5-9fe6-674e69ea840e" (UID: "eaad3e7d-1293-43d5-9fe6-674e69ea840e"). InnerVolumeSpecName "kube-api-access-nm6l2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:34.653331 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:34.653271 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm6l2\" (UniqueName: \"kubernetes.io/projected/eaad3e7d-1293-43d5-9fe6-674e69ea840e-kube-api-access-nm6l2\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:34.765378 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:34.760989 4440 patch_prober.go:29] interesting pod/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": read tcp 10.133.0.2:51232->10.133.0.82:15021: read: connection reset by peer" start-of-body= Oct 31 01:14:34.765378 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:34.761082 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" podUID="583a1183-7c84-40ce-aecb-9663f82a8a47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": read tcp 10.133.0.2:51232->10.133.0.82:15021: read: connection reset by peer" Oct 31 01:14:35.087047 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.087011 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" Oct 31 01:14:35.145277 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.144901 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" Oct 31 01:14:35.162063 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.162027 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/583a1183-7c84-40ce-aecb-9663f82a8a47-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/583a1183-7c84-40ce-aecb-9663f82a8a47-istio-podinfo\") pod \"583a1183-7c84-40ce-aecb-9663f82a8a47\" (UID: \"583a1183-7c84-40ce-aecb-9663f82a8a47\") " Oct 31 01:14:35.162253 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.162131 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/583a1183-7c84-40ce-aecb-9663f82a8a47-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/583a1183-7c84-40ce-aecb-9663f82a8a47-workload-socket\") pod \"583a1183-7c84-40ce-aecb-9663f82a8a47\" (UID: \"583a1183-7c84-40ce-aecb-9663f82a8a47\") " Oct 31 01:14:35.165683 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.162272 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/583a1183-7c84-40ce-aecb-9663f82a8a47-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/583a1183-7c84-40ce-aecb-9663f82a8a47-workload-certs\") pod \"583a1183-7c84-40ce-aecb-9663f82a8a47\" (UID: \"583a1183-7c84-40ce-aecb-9663f82a8a47\") " Oct 31 01:14:35.165683 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.162394 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/583a1183-7c84-40ce-aecb-9663f82a8a47-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/583a1183-7c84-40ce-aecb-9663f82a8a47-istio-envoy\") pod \"583a1183-7c84-40ce-aecb-9663f82a8a47\" (UID: \"583a1183-7c84-40ce-aecb-9663f82a8a47\") " Oct 31 01:14:35.165683 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.162469 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/583a1183-7c84-40ce-aecb-9663f82a8a47-kube-api-access-8zmhr\" (UniqueName: \"kubernetes.io/projected/583a1183-7c84-40ce-aecb-9663f82a8a47-kube-api-access-8zmhr\") pod \"583a1183-7c84-40ce-aecb-9663f82a8a47\" (UID: \"583a1183-7c84-40ce-aecb-9663f82a8a47\") " Oct 31 01:14:35.165683 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.162513 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/583a1183-7c84-40ce-aecb-9663f82a8a47-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/583a1183-7c84-40ce-aecb-9663f82a8a47-credential-socket\") pod \"583a1183-7c84-40ce-aecb-9663f82a8a47\" (UID: \"583a1183-7c84-40ce-aecb-9663f82a8a47\") " Oct 31 01:14:35.165683 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.162565 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/583a1183-7c84-40ce-aecb-9663f82a8a47-istio-token\" (UniqueName: \"kubernetes.io/projected/583a1183-7c84-40ce-aecb-9663f82a8a47-istio-token\") pod \"583a1183-7c84-40ce-aecb-9663f82a8a47\" (UID: \"583a1183-7c84-40ce-aecb-9663f82a8a47\") " Oct 31 01:14:35.165683 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.162622 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/583a1183-7c84-40ce-aecb-9663f82a8a47-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/583a1183-7c84-40ce-aecb-9663f82a8a47-istio-data\") pod \"583a1183-7c84-40ce-aecb-9663f82a8a47\" (UID: \"583a1183-7c84-40ce-aecb-9663f82a8a47\") " Oct 31 01:14:35.165683 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.162667 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/583a1183-7c84-40ce-aecb-9663f82a8a47-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/583a1183-7c84-40ce-aecb-9663f82a8a47-istiod-ca-cert\") pod \"583a1183-7c84-40ce-aecb-9663f82a8a47\" (UID: \"583a1183-7c84-40ce-aecb-9663f82a8a47\") " Oct 31 01:14:35.165683 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.164066 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/583a1183-7c84-40ce-aecb-9663f82a8a47-istiod-ca-cert" pod "583a1183-7c84-40ce-aecb-9663f82a8a47" (UID: "583a1183-7c84-40ce-aecb-9663f82a8a47"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:35.165683 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.165163 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/583a1183-7c84-40ce-aecb-9663f82a8a47-credential-socket" pod "583a1183-7c84-40ce-aecb-9663f82a8a47" (UID: "583a1183-7c84-40ce-aecb-9663f82a8a47"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:35.165683 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.165511 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/583a1183-7c84-40ce-aecb-9663f82a8a47-istio-data" pod "583a1183-7c84-40ce-aecb-9663f82a8a47" (UID: "583a1183-7c84-40ce-aecb-9663f82a8a47"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:35.166373 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.165740 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/583a1183-7c84-40ce-aecb-9663f82a8a47-workload-certs" pod "583a1183-7c84-40ce-aecb-9663f82a8a47" (UID: "583a1183-7c84-40ce-aecb-9663f82a8a47"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:35.166373 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.165794 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/583a1183-7c84-40ce-aecb-9663f82a8a47-workload-socket" pod "583a1183-7c84-40ce-aecb-9663f82a8a47" (UID: "583a1183-7c84-40ce-aecb-9663f82a8a47"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:35.167680 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.167653 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/583a1183-7c84-40ce-aecb-9663f82a8a47-istio-token" pod "583a1183-7c84-40ce-aecb-9663f82a8a47" (UID: "583a1183-7c84-40ce-aecb-9663f82a8a47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:35.168973 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.168933 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/583a1183-7c84-40ce-aecb-9663f82a8a47-istio-podinfo" pod "583a1183-7c84-40ce-aecb-9663f82a8a47" (UID: "583a1183-7c84-40ce-aecb-9663f82a8a47"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:14:35.170116 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.170090 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/583a1183-7c84-40ce-aecb-9663f82a8a47-kube-api-access-8zmhr" pod "583a1183-7c84-40ce-aecb-9663f82a8a47" (UID: "583a1183-7c84-40ce-aecb-9663f82a8a47"). InnerVolumeSpecName "kube-api-access-8zmhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:35.172923 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.172874 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/583a1183-7c84-40ce-aecb-9663f82a8a47-istio-envoy" pod "583a1183-7c84-40ce-aecb-9663f82a8a47" (UID: "583a1183-7c84-40ce-aecb-9663f82a8a47"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:35.214944 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.214907 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" Oct 31 01:14:35.231221 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.231131 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:35.254135 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.254095 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-0a6cca-6599d4945f-dlflj" event={"ID":"eaad3e7d-1293-43d5-9fe6-674e69ea840e","Type":"ContainerDied","Data":"366506dbe164a835a4f265187b14df634d3cad19f2bbe1368078ceb225e71772"} Oct 31 01:14:35.254310 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.254167 4440 scope.go:122] "RemoveContainer" containerID="326d064c31cabc994f3860adf0c6c71f24c1ae6bdcd13f040b76733cd4c943c3" Oct 31 01:14:35.254366 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.254310 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-0a6cca-6599d4945f-dlflj" Oct 31 01:14:35.266677 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.266629 4440 generic.go:356] "Generic (PLEG): container finished" podID="eeb7394c-0c4a-43e6-bbca-f4222bec42ff" containerID="ce2378e90758e17c8ae084350a07551e3ccb018ee1faddd45d9b880089b8a937" exitCode=0 Oct 31 01:14:35.267120 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.266877 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" event={"ID":"eeb7394c-0c4a-43e6-bbca-f4222bec42ff","Type":"ContainerDied","Data":"ce2378e90758e17c8ae084350a07551e3ccb018ee1faddd45d9b880089b8a937"} Oct 31 01:14:35.267341 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.267317 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" event={"ID":"eeb7394c-0c4a-43e6-bbca-f4222bec42ff","Type":"ContainerDied","Data":"ada5233c87952520037e85e3e499fd0e8280c35822c9ec8a84d2c596913a32bb"} Oct 31 01:14:35.267514 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.266967 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq" Oct 31 01:14:35.269742 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.268156 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-istio-token\" (UniqueName: \"kubernetes.io/projected/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-istio-token\") pod \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\" (UID: \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\") " Oct 31 01:14:35.269742 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.268210 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-istiod-ca-cert\") pod \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\" (UID: \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\") " Oct 31 01:14:35.269742 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.268328 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-credential-socket\") pod \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\" (UID: \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\") " Oct 31 01:14:35.269742 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.268363 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-kube-api-access-t75vm\" (UniqueName: \"kubernetes.io/projected/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-kube-api-access-t75vm\") pod \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\" (UID: \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\") " Oct 31 01:14:35.269742 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.268462 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-istio-podinfo\") pod \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\" (UID: \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\") " Oct 31 01:14:35.269742 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.268590 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-workload-certs\") pod \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\" (UID: \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\") " Oct 31 01:14:35.269742 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.268636 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-istio-envoy\") pod \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\" (UID: \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\") " Oct 31 01:14:35.269742 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.268666 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-workload-socket\") pod \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\" (UID: \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\") " Oct 31 01:14:35.269742 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.268699 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-istio-data\") pod \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\" (UID: \"dc34b9f5-85b1-4cab-a49e-c2106d685cc2\") " Oct 31 01:14:35.269742 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.269375 4440 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/583a1183-7c84-40ce-aecb-9663f82a8a47-workload-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.269742 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.269395 4440 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/583a1183-7c84-40ce-aecb-9663f82a8a47-istio-envoy\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.269742 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.269410 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zmhr\" (UniqueName: \"kubernetes.io/projected/583a1183-7c84-40ce-aecb-9663f82a8a47-kube-api-access-8zmhr\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.269742 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.269439 4440 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/583a1183-7c84-40ce-aecb-9663f82a8a47-credential-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.269742 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.269453 4440 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/583a1183-7c84-40ce-aecb-9663f82a8a47-istio-token\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.269742 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.269466 4440 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/583a1183-7c84-40ce-aecb-9663f82a8a47-istio-data\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.269742 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.269481 4440 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/583a1183-7c84-40ce-aecb-9663f82a8a47-istiod-ca-cert\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.269742 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.269495 4440 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/583a1183-7c84-40ce-aecb-9663f82a8a47-istio-podinfo\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.269742 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.269508 4440 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/583a1183-7c84-40ce-aecb-9663f82a8a47-workload-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.271470 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.271150 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-istio-data" pod "dc34b9f5-85b1-4cab-a49e-c2106d685cc2" (UID: "dc34b9f5-85b1-4cab-a49e-c2106d685cc2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:35.273003 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.272980 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-credential-socket" pod "dc34b9f5-85b1-4cab-a49e-c2106d685cc2" (UID: "dc34b9f5-85b1-4cab-a49e-c2106d685cc2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:35.273563 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.273539 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-istiod-ca-cert" pod "dc34b9f5-85b1-4cab-a49e-c2106d685cc2" (UID: "dc34b9f5-85b1-4cab-a49e-c2106d685cc2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:35.274280 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.274160 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-workload-socket" pod "dc34b9f5-85b1-4cab-a49e-c2106d685cc2" (UID: "dc34b9f5-85b1-4cab-a49e-c2106d685cc2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:35.274280 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.274240 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-workload-certs" pod "dc34b9f5-85b1-4cab-a49e-c2106d685cc2" (UID: "dc34b9f5-85b1-4cab-a49e-c2106d685cc2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:35.279637 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.277208 4440 generic.go:356] "Generic (PLEG): container finished" podID="dc34b9f5-85b1-4cab-a49e-c2106d685cc2" containerID="107e6b0b6110a5e9993d7259cfe27fa51563593824c65fc1446ac3ea96c881b7" exitCode=0 Oct 31 01:14:35.279637 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.277345 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" event={"ID":"dc34b9f5-85b1-4cab-a49e-c2106d685cc2","Type":"ContainerDied","Data":"107e6b0b6110a5e9993d7259cfe27fa51563593824c65fc1446ac3ea96c881b7"} Oct 31 01:14:35.279637 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.277385 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" event={"ID":"dc34b9f5-85b1-4cab-a49e-c2106d685cc2","Type":"ContainerDied","Data":"9a921b9aa2d87b245ca7de8ec33b82a3d4bd6834c9528b5f525bf79a89938eec"} Oct 31 01:14:35.279637 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.277499 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6" Oct 31 01:14:35.284626 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.284511 4440 generic.go:356] "Generic (PLEG): container finished" podID="f889944d-e872-4b84-baa6-829de12f3ba1" containerID="7e9c4a7ae854dee3a08c0a09f9e8767db889c6b01e7efb79faa5c8464e615674" exitCode=0 Oct 31 01:14:35.285197 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.285048 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" event={"ID":"f889944d-e872-4b84-baa6-829de12f3ba1","Type":"ContainerDied","Data":"7e9c4a7ae854dee3a08c0a09f9e8767db889c6b01e7efb79faa5c8464e615674"} Oct 31 01:14:35.285197 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.285122 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" event={"ID":"f889944d-e872-4b84-baa6-829de12f3ba1","Type":"ContainerDied","Data":"3c9f1466909b2395696b1b564295edc51efb85d3a8b6a42cca344c7d26562884"} Oct 31 01:14:35.285197 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.285146 4440 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c9f1466909b2395696b1b564295edc51efb85d3a8b6a42cca344c7d26562884" Oct 31 01:14:35.286545 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.286480 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-istio-envoy" pod "dc34b9f5-85b1-4cab-a49e-c2106d685cc2" (UID: "dc34b9f5-85b1-4cab-a49e-c2106d685cc2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:35.290172 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.287656 4440 generic.go:356] "Generic (PLEG): container finished" podID="583a1183-7c84-40ce-aecb-9663f82a8a47" containerID="b6de0db241e1dcd09d56cd8b6d36c57f1a8e0a80d13e136c1b7080cc5d4f5044" exitCode=0 Oct 31 01:14:35.290172 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.287929 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" Oct 31 01:14:35.290172 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.288865 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-kube-api-access-t75vm" pod "dc34b9f5-85b1-4cab-a49e-c2106d685cc2" (UID: "dc34b9f5-85b1-4cab-a49e-c2106d685cc2"). InnerVolumeSpecName "kube-api-access-t75vm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:35.290172 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.288909 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" event={"ID":"583a1183-7c84-40ce-aecb-9663f82a8a47","Type":"ContainerDied","Data":"b6de0db241e1dcd09d56cd8b6d36c57f1a8e0a80d13e136c1b7080cc5d4f5044"} Oct 31 01:14:35.290172 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.288961 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts" event={"ID":"583a1183-7c84-40ce-aecb-9663f82a8a47","Type":"ContainerDied","Data":"32ae1d149f889e62f5298f82948d6c6f7257f04f5a97a173fe68b1c53193ffd5"} Oct 31 01:14:35.308683 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.308652 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" Oct 31 01:14:35.310524 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.310485 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-istio-token" pod "dc34b9f5-85b1-4cab-a49e-c2106d685cc2" (UID: "dc34b9f5-85b1-4cab-a49e-c2106d685cc2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:35.317411 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.316306 4440 scope.go:122] "RemoveContainer" containerID="ce2378e90758e17c8ae084350a07551e3ccb018ee1faddd45d9b880089b8a937" Oct 31 01:14:35.317411 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.316661 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-istio-podinfo" pod "dc34b9f5-85b1-4cab-a49e-c2106d685cc2" (UID: "dc34b9f5-85b1-4cab-a49e-c2106d685cc2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:14:35.360392 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.359483 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-0a6cca-6599d4945f-dlflj" podUID="eaad3e7d-1293-43d5-9fe6-674e69ea840e" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.60:50052\" within 1s: context deadline exceeded" Oct 31 01:14:35.360392 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.360050 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-0a6cca-6599d4945f-dlflj" Oct 31 01:14:35.361312 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:14:35.360693 4440 logging.go:55] [core] [Channel #887 SubChannel #888]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.60:50052", ServerName: "10.133.0.60:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.60:50052: operation was canceled" Oct 31 01:14:35.372779 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.372735 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:35.373274 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.373249 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-istiod-ca-cert\") pod \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\" (UID: \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\") " Oct 31 01:14:35.374412 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.374387 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-kube-api-access-bk6vb\" (UniqueName: \"kubernetes.io/projected/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-kube-api-access-bk6vb\") pod \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\" (UID: \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\") " Oct 31 01:14:35.376050 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.376026 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-workload-certs\") pod \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\" (UID: \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\") " Oct 31 01:14:35.376246 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.376232 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-istio-token\" (UniqueName: \"kubernetes.io/projected/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-istio-token\") pod \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\" (UID: \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\") " Oct 31 01:14:35.376429 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.376404 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-credential-socket\") pod \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\" (UID: \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\") " Oct 31 01:14:35.376616 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.376604 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-istio-envoy\") pod \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\" (UID: \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\") " Oct 31 01:14:35.376734 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.376721 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-istio-podinfo\") pod \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\" (UID: \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\") " Oct 31 01:14:35.376855 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.376845 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-workload-socket\") pod \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\" (UID: \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\") " Oct 31 01:14:35.376948 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.376938 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-istio-data\") pod \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\" (UID: \"eeb7394c-0c4a-43e6-bbca-f4222bec42ff\") " Oct 31 01:14:35.377792 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.377743 4440 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-istiod-ca-cert\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.377892 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.377879 4440 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-credential-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.377965 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.377954 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t75vm\" (UniqueName: \"kubernetes.io/projected/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-kube-api-access-t75vm\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.378024 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.378016 4440 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-istio-podinfo\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.378091 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.378081 4440 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-workload-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.378153 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.378145 4440 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-istio-envoy\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.378209 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.378201 4440 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-workload-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.378314 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.378305 4440 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-istio-data\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.378377 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.378367 4440 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc34b9f5-85b1-4cab-a49e-c2106d685cc2-istio-token\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.389484 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.388596 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-workload-certs" pod "eeb7394c-0c4a-43e6-bbca-f4222bec42ff" (UID: "eeb7394c-0c4a-43e6-bbca-f4222bec42ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:35.392585 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.392551 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-credential-socket" pod "eeb7394c-0c4a-43e6-bbca-f4222bec42ff" (UID: "eeb7394c-0c4a-43e6-bbca-f4222bec42ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:35.393514 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.393182 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-istiod-ca-cert" pod "eeb7394c-0c4a-43e6-bbca-f4222bec42ff" (UID: "eeb7394c-0c4a-43e6-bbca-f4222bec42ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:35.394178 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.394098 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-workload-socket" pod "eeb7394c-0c4a-43e6-bbca-f4222bec42ff" (UID: "eeb7394c-0c4a-43e6-bbca-f4222bec42ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:35.403478 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.403440 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-istio-data" pod "eeb7394c-0c4a-43e6-bbca-f4222bec42ff" (UID: "eeb7394c-0c4a-43e6-bbca-f4222bec42ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:35.416092 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.416055 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-kube-api-access-bk6vb" pod "eeb7394c-0c4a-43e6-bbca-f4222bec42ff" (UID: "eeb7394c-0c4a-43e6-bbca-f4222bec42ff"). InnerVolumeSpecName "kube-api-access-bk6vb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:35.417049 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.417012 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-istio-token" pod "eeb7394c-0c4a-43e6-bbca-f4222bec42ff" (UID: "eeb7394c-0c4a-43e6-bbca-f4222bec42ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:35.419611 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.419096 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-istio-podinfo" pod "eeb7394c-0c4a-43e6-bbca-f4222bec42ff" (UID: "eeb7394c-0c4a-43e6-bbca-f4222bec42ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:14:35.420368 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.420336 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-istio-envoy" pod "eeb7394c-0c4a-43e6-bbca-f4222bec42ff" (UID: "eeb7394c-0c4a-43e6-bbca-f4222bec42ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:35.423882 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.423772 4440 scope.go:122] "RemoveContainer" containerID="ce2378e90758e17c8ae084350a07551e3ccb018ee1faddd45d9b880089b8a937" Oct 31 01:14:35.427011 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:14:35.426975 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce2378e90758e17c8ae084350a07551e3ccb018ee1faddd45d9b880089b8a937\": container with ID starting with ce2378e90758e17c8ae084350a07551e3ccb018ee1faddd45d9b880089b8a937 not found: ID does not exist" containerID="ce2378e90758e17c8ae084350a07551e3ccb018ee1faddd45d9b880089b8a937" Oct 31 01:14:35.427820 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.427784 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce2378e90758e17c8ae084350a07551e3ccb018ee1faddd45d9b880089b8a937"} err="failed to get container status \"ce2378e90758e17c8ae084350a07551e3ccb018ee1faddd45d9b880089b8a937\": rpc error: code = NotFound desc = could not find container \"ce2378e90758e17c8ae084350a07551e3ccb018ee1faddd45d9b880089b8a937\": container with ID starting with ce2378e90758e17c8ae084350a07551e3ccb018ee1faddd45d9b880089b8a937 not found: ID does not exist" Oct 31 01:14:35.427987 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.427961 4440 scope.go:122] "RemoveContainer" containerID="107e6b0b6110a5e9993d7259cfe27fa51563593824c65fc1446ac3ea96c881b7" Oct 31 01:14:35.435472 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.434194 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-0a6cca-6599d4945f-dlflj"] Oct 31 01:14:35.436093 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.436070 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-0a6cca-6599d4945f-dlflj"] Oct 31 01:14:35.443211 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.443176 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts"] Oct 31 01:14:35.451051 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.450143 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-bggts"] Oct 31 01:14:35.471763 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.471732 4440 scope.go:122] "RemoveContainer" containerID="107e6b0b6110a5e9993d7259cfe27fa51563593824c65fc1446ac3ea96c881b7" Oct 31 01:14:35.474668 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:14:35.474480 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"107e6b0b6110a5e9993d7259cfe27fa51563593824c65fc1446ac3ea96c881b7\": container with ID starting with 107e6b0b6110a5e9993d7259cfe27fa51563593824c65fc1446ac3ea96c881b7 not found: ID does not exist" containerID="107e6b0b6110a5e9993d7259cfe27fa51563593824c65fc1446ac3ea96c881b7" Oct 31 01:14:35.474668 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.474522 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"107e6b0b6110a5e9993d7259cfe27fa51563593824c65fc1446ac3ea96c881b7"} err="failed to get container status \"107e6b0b6110a5e9993d7259cfe27fa51563593824c65fc1446ac3ea96c881b7\": rpc error: code = NotFound desc = could not find container \"107e6b0b6110a5e9993d7259cfe27fa51563593824c65fc1446ac3ea96c881b7\": container with ID starting with 107e6b0b6110a5e9993d7259cfe27fa51563593824c65fc1446ac3ea96c881b7 not found: ID does not exist" Oct 31 01:14:35.474668 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.474548 4440 scope.go:122] "RemoveContainer" containerID="b6de0db241e1dcd09d56cd8b6d36c57f1a8e0a80d13e136c1b7080cc5d4f5044" Oct 31 01:14:35.497779 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.497747 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f889944d-e872-4b84-baa6-829de12f3ba1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f889944d-e872-4b84-baa6-829de12f3ba1-workload-certs\") pod \"f889944d-e872-4b84-baa6-829de12f3ba1\" (UID: \"f889944d-e872-4b84-baa6-829de12f3ba1\") " Oct 31 01:14:35.498111 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.498084 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f889944d-e872-4b84-baa6-829de12f3ba1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f889944d-e872-4b84-baa6-829de12f3ba1-istiod-ca-cert\") pod \"f889944d-e872-4b84-baa6-829de12f3ba1\" (UID: \"f889944d-e872-4b84-baa6-829de12f3ba1\") " Oct 31 01:14:35.498360 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.498341 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f889944d-e872-4b84-baa6-829de12f3ba1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f889944d-e872-4b84-baa6-829de12f3ba1-istio-envoy\") pod \"f889944d-e872-4b84-baa6-829de12f3ba1\" (UID: \"f889944d-e872-4b84-baa6-829de12f3ba1\") " Oct 31 01:14:35.498638 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.498580 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f889944d-e872-4b84-baa6-829de12f3ba1-istio-token\" (UniqueName: \"kubernetes.io/projected/f889944d-e872-4b84-baa6-829de12f3ba1-istio-token\") pod \"f889944d-e872-4b84-baa6-829de12f3ba1\" (UID: \"f889944d-e872-4b84-baa6-829de12f3ba1\") " Oct 31 01:14:35.498804 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.498785 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f889944d-e872-4b84-baa6-829de12f3ba1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f889944d-e872-4b84-baa6-829de12f3ba1-credential-socket\") pod \"f889944d-e872-4b84-baa6-829de12f3ba1\" (UID: \"f889944d-e872-4b84-baa6-829de12f3ba1\") " Oct 31 01:14:35.498941 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.498925 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f889944d-e872-4b84-baa6-829de12f3ba1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f889944d-e872-4b84-baa6-829de12f3ba1-workload-socket\") pod \"f889944d-e872-4b84-baa6-829de12f3ba1\" (UID: \"f889944d-e872-4b84-baa6-829de12f3ba1\") " Oct 31 01:14:35.499259 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.499239 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f889944d-e872-4b84-baa6-829de12f3ba1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f889944d-e872-4b84-baa6-829de12f3ba1-istio-podinfo\") pod \"f889944d-e872-4b84-baa6-829de12f3ba1\" (UID: \"f889944d-e872-4b84-baa6-829de12f3ba1\") " Oct 31 01:14:35.499395 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.499379 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f889944d-e872-4b84-baa6-829de12f3ba1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f889944d-e872-4b84-baa6-829de12f3ba1-istio-data\") pod \"f889944d-e872-4b84-baa6-829de12f3ba1\" (UID: \"f889944d-e872-4b84-baa6-829de12f3ba1\") " Oct 31 01:14:35.499601 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.499581 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f889944d-e872-4b84-baa6-829de12f3ba1-kube-api-access-9586g\" (UniqueName: \"kubernetes.io/projected/f889944d-e872-4b84-baa6-829de12f3ba1-kube-api-access-9586g\") pod \"f889944d-e872-4b84-baa6-829de12f3ba1\" (UID: \"f889944d-e872-4b84-baa6-829de12f3ba1\") " Oct 31 01:14:35.500549 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.500528 4440 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-istio-envoy\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.500682 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.500664 4440 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-istio-podinfo\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.500779 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.500766 4440 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-workload-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.500870 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.500858 4440 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-istio-data\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.500969 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.500956 4440 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-istiod-ca-cert\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.501061 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.501047 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk6vb\" (UniqueName: \"kubernetes.io/projected/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-kube-api-access-bk6vb\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.501149 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.501137 4440 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-workload-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.501242 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.501230 4440 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-istio-token\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.501334 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.501321 4440 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeb7394c-0c4a-43e6-bbca-f4222bec42ff-credential-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.509539 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.509475 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f889944d-e872-4b84-baa6-829de12f3ba1-credential-socket" pod "f889944d-e872-4b84-baa6-829de12f3ba1" (UID: "f889944d-e872-4b84-baa6-829de12f3ba1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:35.510052 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.510003 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f889944d-e872-4b84-baa6-829de12f3ba1-workload-socket" pod "f889944d-e872-4b84-baa6-829de12f3ba1" (UID: "f889944d-e872-4b84-baa6-829de12f3ba1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:35.517341 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.516670 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f889944d-e872-4b84-baa6-829de12f3ba1-istiod-ca-cert" pod "f889944d-e872-4b84-baa6-829de12f3ba1" (UID: "f889944d-e872-4b84-baa6-829de12f3ba1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:35.517920 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.517582 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f889944d-e872-4b84-baa6-829de12f3ba1-workload-certs" pod "f889944d-e872-4b84-baa6-829de12f3ba1" (UID: "f889944d-e872-4b84-baa6-829de12f3ba1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:35.518647 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.518622 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f889944d-e872-4b84-baa6-829de12f3ba1-istio-data" pod "f889944d-e872-4b84-baa6-829de12f3ba1" (UID: "f889944d-e872-4b84-baa6-829de12f3ba1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:35.519777 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.519645 4440 scope.go:122] "RemoveContainer" containerID="b6de0db241e1dcd09d56cd8b6d36c57f1a8e0a80d13e136c1b7080cc5d4f5044" Oct 31 01:14:35.520237 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:14:35.520168 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6de0db241e1dcd09d56cd8b6d36c57f1a8e0a80d13e136c1b7080cc5d4f5044\": container with ID starting with b6de0db241e1dcd09d56cd8b6d36c57f1a8e0a80d13e136c1b7080cc5d4f5044 not found: ID does not exist" containerID="b6de0db241e1dcd09d56cd8b6d36c57f1a8e0a80d13e136c1b7080cc5d4f5044" Oct 31 01:14:35.520237 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.520207 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6de0db241e1dcd09d56cd8b6d36c57f1a8e0a80d13e136c1b7080cc5d4f5044"} err="failed to get container status \"b6de0db241e1dcd09d56cd8b6d36c57f1a8e0a80d13e136c1b7080cc5d4f5044\": rpc error: code = NotFound desc = could not find container \"b6de0db241e1dcd09d56cd8b6d36c57f1a8e0a80d13e136c1b7080cc5d4f5044\": container with ID starting with b6de0db241e1dcd09d56cd8b6d36c57f1a8e0a80d13e136c1b7080cc5d4f5044 not found: ID does not exist" Oct 31 01:14:35.520961 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.520940 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f889944d-e872-4b84-baa6-829de12f3ba1-kube-api-access-9586g" pod "f889944d-e872-4b84-baa6-829de12f3ba1" (UID: "f889944d-e872-4b84-baa6-829de12f3ba1"). InnerVolumeSpecName "kube-api-access-9586g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:35.521206 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.521149 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f889944d-e872-4b84-baa6-829de12f3ba1-istio-podinfo" pod "f889944d-e872-4b84-baa6-829de12f3ba1" (UID: "f889944d-e872-4b84-baa6-829de12f3ba1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:14:35.523845 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.523810 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f889944d-e872-4b84-baa6-829de12f3ba1-istio-token" pod "f889944d-e872-4b84-baa6-829de12f3ba1" (UID: "f889944d-e872-4b84-baa6-829de12f3ba1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:35.525643 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.525606 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f889944d-e872-4b84-baa6-829de12f3ba1-istio-envoy" pod "f889944d-e872-4b84-baa6-829de12f3ba1" (UID: "f889944d-e872-4b84-baa6-829de12f3ba1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:35.578849 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.578748 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-252485f4-f4rvq" Oct 31 01:14:35.603233 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.602331 4440 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f889944d-e872-4b84-baa6-829de12f3ba1-istio-token\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.603233 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.602363 4440 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f889944d-e872-4b84-baa6-829de12f3ba1-credential-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.603233 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.602373 4440 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f889944d-e872-4b84-baa6-829de12f3ba1-workload-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.603233 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.602382 4440 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f889944d-e872-4b84-baa6-829de12f3ba1-istio-podinfo\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.603233 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.602391 4440 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f889944d-e872-4b84-baa6-829de12f3ba1-istio-data\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.603233 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.602399 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9586g\" (UniqueName: \"kubernetes.io/projected/f889944d-e872-4b84-baa6-829de12f3ba1-kube-api-access-9586g\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.603233 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.602409 4440 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f889944d-e872-4b84-baa6-829de12f3ba1-workload-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.610870 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.605898 4440 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f889944d-e872-4b84-baa6-829de12f3ba1-istiod-ca-cert\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.610870 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.605939 4440 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f889944d-e872-4b84-baa6-829de12f3ba1-istio-envoy\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.645520 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.645413 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq"] Oct 31 01:14:35.650869 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.650833 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-k7pkq"] Oct 31 01:14:35.666880 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.666847 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6"] Oct 31 01:14:35.674887 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.674782 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-w28w6"] Oct 31 01:14:35.708671 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.707397 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f77962d-e14f-4225-b2e9-7a709dc8c4f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f77962d-e14f-4225-b2e9-7a709dc8c4f4-maas-api-openshift-ca\") pod \"7f77962d-e14f-4225-b2e9-7a709dc8c4f4\" (UID: \"7f77962d-e14f-4225-b2e9-7a709dc8c4f4\") " Oct 31 01:14:35.708671 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.707609 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f77962d-e14f-4225-b2e9-7a709dc8c4f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f77962d-e14f-4225-b2e9-7a709dc8c4f4-maas-api-xks-ca\") pod \"7f77962d-e14f-4225-b2e9-7a709dc8c4f4\" (UID: \"7f77962d-e14f-4225-b2e9-7a709dc8c4f4\") " Oct 31 01:14:35.708671 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.707780 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f77962d-e14f-4225-b2e9-7a709dc8c4f4-kube-api-access-nsr9t\" (UniqueName: \"kubernetes.io/projected/7f77962d-e14f-4225-b2e9-7a709dc8c4f4-kube-api-access-nsr9t\") pod \"7f77962d-e14f-4225-b2e9-7a709dc8c4f4\" (UID: \"7f77962d-e14f-4225-b2e9-7a709dc8c4f4\") " Oct 31 01:14:35.708671 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.707806 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f77962d-e14f-4225-b2e9-7a709dc8c4f4-maas-api-openshift-ca" pod "7f77962d-e14f-4225-b2e9-7a709dc8c4f4" (UID: "7f77962d-e14f-4225-b2e9-7a709dc8c4f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:35.708671 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.708629 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f77962d-e14f-4225-b2e9-7a709dc8c4f4-maas-api-openshift-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.712077 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.712044 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f77962d-e14f-4225-b2e9-7a709dc8c4f4-kube-api-access-nsr9t" pod "7f77962d-e14f-4225-b2e9-7a709dc8c4f4" (UID: "7f77962d-e14f-4225-b2e9-7a709dc8c4f4"). InnerVolumeSpecName "kube-api-access-nsr9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:35.712077 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.712060 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f77962d-e14f-4225-b2e9-7a709dc8c4f4-maas-api-xks-ca" pod "7f77962d-e14f-4225-b2e9-7a709dc8c4f4" (UID: "7f77962d-e14f-4225-b2e9-7a709dc8c4f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:14:35.776719 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.776685 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="583a1183-7c84-40ce-aecb-9663f82a8a47" path="/var/lib/kubelet/pods/583a1183-7c84-40ce-aecb-9663f82a8a47/volumes" Oct 31 01:14:35.777309 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.777291 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc34b9f5-85b1-4cab-a49e-c2106d685cc2" path="/var/lib/kubelet/pods/dc34b9f5-85b1-4cab-a49e-c2106d685cc2/volumes" Oct 31 01:14:35.777787 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.777772 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaad3e7d-1293-43d5-9fe6-674e69ea840e" path="/var/lib/kubelet/pods/eaad3e7d-1293-43d5-9fe6-674e69ea840e/volumes" Oct 31 01:14:35.778591 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.778568 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eeb7394c-0c4a-43e6-bbca-f4222bec42ff" path="/var/lib/kubelet/pods/eeb7394c-0c4a-43e6-bbca-f4222bec42ff/volumes" Oct 31 01:14:35.810115 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.810073 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsr9t\" (UniqueName: \"kubernetes.io/projected/7f77962d-e14f-4225-b2e9-7a709dc8c4f4-kube-api-access-nsr9t\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.810115 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.810115 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f77962d-e14f-4225-b2e9-7a709dc8c4f4-maas-api-xks-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:35.905716 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:35.905618 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-28ff08-8947d86b5-frcjf" podUID="b8df1e49-cfaf-4c8c-9498-3400501ae86b" containerName="payload-processing" containerID="cri-o://132e471216ed1fe2da347dc8ccaf1379ba6bac80ff6fd23338cbd50509a26a0c" gracePeriod=25 Oct 31 01:14:36.241789 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:36.241664 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:36.311937 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:36.310969 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-252485f4-f4rvq" event={"ID":"7f77962d-e14f-4225-b2e9-7a709dc8c4f4","Type":"ContainerDied","Data":"0e3563408a061f616a3546fcfca286a9041e5cc9434b9f36c71e35ddcbba6235"} Oct 31 01:14:36.311937 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:36.311040 4440 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e3563408a061f616a3546fcfca286a9041e5cc9434b9f36c71e35ddcbba6235" Oct 31 01:14:36.311937 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:36.311105 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-252485f4-f4rvq" Oct 31 01:14:36.327059 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:36.324043 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5" Oct 31 01:14:36.369555 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:36.368930 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5"] Oct 31 01:14:36.377091 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:36.377033 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-0a6cca-openshift-default-7cf4f9647c-g85n5"] Oct 31 01:14:37.064369 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:37.064286 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:37.380687 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:37.357405 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:14:37.380687 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:37.357486 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:14:37.383505 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:37.383469 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-28ff08-8947d86b5-frcjf" podUID="b8df1e49-cfaf-4c8c-9498-3400501ae86b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:14:37.501266 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:37.499338 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj"] Oct 31 01:14:37.501266 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:37.499632 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj" podUID="be67bf17-2e3c-4624-96cd-aab05f0cdc27" containerName="maas-api" containerID="cri-o://cf5a35c93de6f2c77d63bd35bd61db926d573cc99f29fb531007f0d3f9fb65ae" gracePeriod=30 Oct 31 01:14:37.565274 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:37.565208 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:37.592600 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:37.592562 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj" podUID="be67bf17-2e3c-4624-96cd-aab05f0cdc27" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.67:8443/health\": dial tcp 10.133.0.67:8443: connect: connection refused" Oct 31 01:14:37.690723 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:37.690679 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:37.937186 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:37.937144 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f889944d-e872-4b84-baa6-829de12f3ba1" path="/var/lib/kubelet/pods/f889944d-e872-4b84-baa6-829de12f3ba1/volumes" Oct 31 01:14:37.945493 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:37.944131 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:38.074179 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:38.074116 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:38.328150 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:38.328043 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:38.345324 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:38.345283 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkx55" event={"ID":"4f77d13d-b9ab-44cd-ba25-37f8841cf8da","Type":"ContainerStarted","Data":"c9c567a30c45fafc6d3e24ed529167e2d7f85125456380f100d331f16fa7c756"} Oct 31 01:14:38.345506 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:38.345351 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkx55" Oct 31 01:14:38.347783 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:38.347748 4440 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-bkx55 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:14:38.347926 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:38.347811 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkx55" podUID="4f77d13d-b9ab-44cd-ba25-37f8841cf8da" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:14:38.369698 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:38.369644 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkx55" podStartSLOduration=2.191589512 podStartE2EDuration="6.369626742s" totalImagesPullingTime="4.17803723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:14:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:14:38.364894951 +0000 UTC m=+1122.723617269" watchObservedRunningTime="2026-10-31 01:14:38.369626742 +0000 UTC m=+1122.728349057" Oct 31 01:14:38.767957 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:38.767920 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:38.877459 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:38.877364 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj" Oct 31 01:14:38.878329 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:38.878300 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:38.933465 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:38.933330 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be67bf17-2e3c-4624-96cd-aab05f0cdc27-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be67bf17-2e3c-4624-96cd-aab05f0cdc27-maas-api-tls\") pod \"be67bf17-2e3c-4624-96cd-aab05f0cdc27\" (UID: \"be67bf17-2e3c-4624-96cd-aab05f0cdc27\") " Oct 31 01:14:38.933670 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:38.933528 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be67bf17-2e3c-4624-96cd-aab05f0cdc27-kube-api-access-25r9x\" (UniqueName: \"kubernetes.io/projected/be67bf17-2e3c-4624-96cd-aab05f0cdc27-kube-api-access-25r9x\") pod \"be67bf17-2e3c-4624-96cd-aab05f0cdc27\" (UID: \"be67bf17-2e3c-4624-96cd-aab05f0cdc27\") " Oct 31 01:14:38.933670 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:38.933589 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be67bf17-2e3c-4624-96cd-aab05f0cdc27-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be67bf17-2e3c-4624-96cd-aab05f0cdc27-openshift-service-ca\") pod \"be67bf17-2e3c-4624-96cd-aab05f0cdc27\" (UID: \"be67bf17-2e3c-4624-96cd-aab05f0cdc27\") " Oct 31 01:14:38.933815 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:38.933801 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be67bf17-2e3c-4624-96cd-aab05f0cdc27-metrics-certs\" (UniqueName: \"kubernetes.io/secret/be67bf17-2e3c-4624-96cd-aab05f0cdc27-metrics-certs\") pod \"be67bf17-2e3c-4624-96cd-aab05f0cdc27\" (UID: \"be67bf17-2e3c-4624-96cd-aab05f0cdc27\") " Oct 31 01:14:38.935136 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:38.935103 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be67bf17-2e3c-4624-96cd-aab05f0cdc27-openshift-service-ca" pod "be67bf17-2e3c-4624-96cd-aab05f0cdc27" (UID: "be67bf17-2e3c-4624-96cd-aab05f0cdc27"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:38.938963 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:38.937707 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be67bf17-2e3c-4624-96cd-aab05f0cdc27-maas-api-tls" pod "be67bf17-2e3c-4624-96cd-aab05f0cdc27" (UID: "be67bf17-2e3c-4624-96cd-aab05f0cdc27"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:14:38.938963 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:38.938858 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be67bf17-2e3c-4624-96cd-aab05f0cdc27-kube-api-access-25r9x" pod "be67bf17-2e3c-4624-96cd-aab05f0cdc27" (UID: "be67bf17-2e3c-4624-96cd-aab05f0cdc27"). InnerVolumeSpecName "kube-api-access-25r9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:38.939562 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:38.939527 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be67bf17-2e3c-4624-96cd-aab05f0cdc27-metrics-certs" pod "be67bf17-2e3c-4624-96cd-aab05f0cdc27" (UID: "be67bf17-2e3c-4624-96cd-aab05f0cdc27"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:14:39.035809 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:39.035763 4440 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be67bf17-2e3c-4624-96cd-aab05f0cdc27-metrics-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:39.035809 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:39.035800 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be67bf17-2e3c-4624-96cd-aab05f0cdc27-maas-api-tls\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:39.035809 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:39.035810 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25r9x\" (UniqueName: \"kubernetes.io/projected/be67bf17-2e3c-4624-96cd-aab05f0cdc27-kube-api-access-25r9x\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:39.035809 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:39.035821 4440 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be67bf17-2e3c-4624-96cd-aab05f0cdc27-openshift-service-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:39.051227 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:39.051184 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:39.355817 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:39.355776 4440 generic.go:356] "Generic (PLEG): container finished" podID="be67bf17-2e3c-4624-96cd-aab05f0cdc27" containerID="cf5a35c93de6f2c77d63bd35bd61db926d573cc99f29fb531007f0d3f9fb65ae" exitCode=0 Oct 31 01:14:39.356066 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:39.355822 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj" Oct 31 01:14:39.356066 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:39.355858 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj" event={"ID":"be67bf17-2e3c-4624-96cd-aab05f0cdc27","Type":"ContainerDied","Data":"cf5a35c93de6f2c77d63bd35bd61db926d573cc99f29fb531007f0d3f9fb65ae"} Oct 31 01:14:39.356066 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:39.355916 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj" event={"ID":"be67bf17-2e3c-4624-96cd-aab05f0cdc27","Type":"ContainerDied","Data":"023668299ce4f37e209367d12bd324d86b051604d890f81ce338f38cee8da3c1"} Oct 31 01:14:39.356066 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:39.355942 4440 scope.go:122] "RemoveContainer" containerID="cf5a35c93de6f2c77d63bd35bd61db926d573cc99f29fb531007f0d3f9fb65ae" Oct 31 01:14:39.357299 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:39.357282 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkx55" Oct 31 01:14:39.368762 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:39.368743 4440 scope.go:122] "RemoveContainer" containerID="cf5a35c93de6f2c77d63bd35bd61db926d573cc99f29fb531007f0d3f9fb65ae" Oct 31 01:14:39.369098 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:14:39.369077 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf5a35c93de6f2c77d63bd35bd61db926d573cc99f29fb531007f0d3f9fb65ae\": container with ID starting with cf5a35c93de6f2c77d63bd35bd61db926d573cc99f29fb531007f0d3f9fb65ae not found: ID does not exist" containerID="cf5a35c93de6f2c77d63bd35bd61db926d573cc99f29fb531007f0d3f9fb65ae" Oct 31 01:14:39.369162 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:39.369110 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf5a35c93de6f2c77d63bd35bd61db926d573cc99f29fb531007f0d3f9fb65ae"} err="failed to get container status \"cf5a35c93de6f2c77d63bd35bd61db926d573cc99f29fb531007f0d3f9fb65ae\": rpc error: code = NotFound desc = could not find container \"cf5a35c93de6f2c77d63bd35bd61db926d573cc99f29fb531007f0d3f9fb65ae\": container with ID starting with cf5a35c93de6f2c77d63bd35bd61db926d573cc99f29fb531007f0d3f9fb65ae not found: ID does not exist" Oct 31 01:14:39.401242 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:39.401202 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj"] Oct 31 01:14:39.436468 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:39.425490 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8499c8-655ffcbd76-6hxxj"] Oct 31 01:14:39.475280 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:39.475237 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8499c8-5c65ddd74c-9r5n8" podUID="ab8ba016-584a-4646-a3d4-f6db4ccf0700" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:14:39.773170 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:39.773134 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be67bf17-2e3c-4624-96cd-aab05f0cdc27" path="/var/lib/kubelet/pods/be67bf17-2e3c-4624-96cd-aab05f0cdc27/volumes" Oct 31 01:14:39.910384 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:39.910349 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:40.354787 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.350923 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" containerID="cri-o://c3ff32cc0a574168dc30068c8dcc6454aaa719461d84018cce68768fdc6b1b78" gracePeriod=45 Oct 31 01:14:40.369394 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.366968 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" podUID="cf86081d-7942-454a-8763-f658bd0ce870" containerName="main" containerID="cri-o://c4efb30afb1028e371b427a798ca232592bde14d3173fc46f372a61d911f535f" gracePeriod=45 Oct 31 01:14:40.398260 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.398218 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" podUID="cf86081d-7942-454a-8763-f658bd0ce870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": dial tcp 10.133.0.90:8000: connect: connection refused" Oct 31 01:14:40.483339 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.483303 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:40.652298 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.648948 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:40.767720 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.767684 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" Oct 31 01:14:40.770576 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.770319 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" Oct 31 01:14:40.876031 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.875929 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf86081d-7942-454a-8763-f658bd0ce870-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf86081d-7942-454a-8763-f658bd0ce870-dshm\") pod \"cf86081d-7942-454a-8763-f658bd0ce870\" (UID: \"cf86081d-7942-454a-8763-f658bd0ce870\") " Oct 31 01:14:40.876554 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.876034 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f93d4671-a358-4547-ba77-06255ca42c3e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f93d4671-a358-4547-ba77-06255ca42c3e-model-cache\") pod \"f93d4671-a358-4547-ba77-06255ca42c3e\" (UID: \"f93d4671-a358-4547-ba77-06255ca42c3e\") " Oct 31 01:14:40.876554 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.876093 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf86081d-7942-454a-8763-f658bd0ce870-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf86081d-7942-454a-8763-f658bd0ce870-tls-certs\") pod \"cf86081d-7942-454a-8763-f658bd0ce870\" (UID: \"cf86081d-7942-454a-8763-f658bd0ce870\") " Oct 31 01:14:40.876554 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.876139 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f93d4671-a358-4547-ba77-06255ca42c3e-tls-certs\" (UniqueName: \"kubernetes.io/secret/f93d4671-a358-4547-ba77-06255ca42c3e-tls-certs\") pod \"f93d4671-a358-4547-ba77-06255ca42c3e\" (UID: \"f93d4671-a358-4547-ba77-06255ca42c3e\") " Oct 31 01:14:40.876554 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.876174 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf86081d-7942-454a-8763-f658bd0ce870-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf86081d-7942-454a-8763-f658bd0ce870-model-cache\") pod \"cf86081d-7942-454a-8763-f658bd0ce870\" (UID: \"cf86081d-7942-454a-8763-f658bd0ce870\") " Oct 31 01:14:40.876554 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.876222 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f93d4671-a358-4547-ba77-06255ca42c3e-kube-api-access-qk4mt\" (UniqueName: \"kubernetes.io/projected/f93d4671-a358-4547-ba77-06255ca42c3e-kube-api-access-qk4mt\") pod \"f93d4671-a358-4547-ba77-06255ca42c3e\" (UID: \"f93d4671-a358-4547-ba77-06255ca42c3e\") " Oct 31 01:14:40.876554 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.876276 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f93d4671-a358-4547-ba77-06255ca42c3e-model-cache" pod "f93d4671-a358-4547-ba77-06255ca42c3e" (UID: "f93d4671-a358-4547-ba77-06255ca42c3e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:40.876554 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.876294 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f93d4671-a358-4547-ba77-06255ca42c3e-home\" (UniqueName: \"kubernetes.io/empty-dir/f93d4671-a358-4547-ba77-06255ca42c3e-home\") pod \"f93d4671-a358-4547-ba77-06255ca42c3e\" (UID: \"f93d4671-a358-4547-ba77-06255ca42c3e\") " Oct 31 01:14:40.876554 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.876357 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f93d4671-a358-4547-ba77-06255ca42c3e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f93d4671-a358-4547-ba77-06255ca42c3e-dshm\") pod \"f93d4671-a358-4547-ba77-06255ca42c3e\" (UID: \"f93d4671-a358-4547-ba77-06255ca42c3e\") " Oct 31 01:14:40.876554 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.876469 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf86081d-7942-454a-8763-f658bd0ce870-model-cache" pod "cf86081d-7942-454a-8763-f658bd0ce870" (UID: "cf86081d-7942-454a-8763-f658bd0ce870"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:40.876554 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.876500 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf86081d-7942-454a-8763-f658bd0ce870-kube-api-access-5d2jj\" (UniqueName: \"kubernetes.io/projected/cf86081d-7942-454a-8763-f658bd0ce870-kube-api-access-5d2jj\") pod \"cf86081d-7942-454a-8763-f658bd0ce870\" (UID: \"cf86081d-7942-454a-8763-f658bd0ce870\") " Oct 31 01:14:40.876554 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.876558 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f93d4671-a358-4547-ba77-06255ca42c3e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f93d4671-a358-4547-ba77-06255ca42c3e-tmp-dir\") pod \"f93d4671-a358-4547-ba77-06255ca42c3e\" (UID: \"f93d4671-a358-4547-ba77-06255ca42c3e\") " Oct 31 01:14:40.877093 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.876649 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf86081d-7942-454a-8763-f658bd0ce870-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf86081d-7942-454a-8763-f658bd0ce870-tmp-dir\") pod \"cf86081d-7942-454a-8763-f658bd0ce870\" (UID: \"cf86081d-7942-454a-8763-f658bd0ce870\") " Oct 31 01:14:40.877093 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.876673 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f93d4671-a358-4547-ba77-06255ca42c3e-home" pod "f93d4671-a358-4547-ba77-06255ca42c3e" (UID: "f93d4671-a358-4547-ba77-06255ca42c3e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:40.877093 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.876677 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf86081d-7942-454a-8763-f658bd0ce870-home\" (UniqueName: \"kubernetes.io/empty-dir/cf86081d-7942-454a-8763-f658bd0ce870-home\") pod \"cf86081d-7942-454a-8763-f658bd0ce870\" (UID: \"cf86081d-7942-454a-8763-f658bd0ce870\") " Oct 31 01:14:40.877093 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.877067 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f93d4671-a358-4547-ba77-06255ca42c3e-tmp-dir" pod "f93d4671-a358-4547-ba77-06255ca42c3e" (UID: "f93d4671-a358-4547-ba77-06255ca42c3e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:40.877338 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.877195 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf86081d-7942-454a-8763-f658bd0ce870-home" pod "cf86081d-7942-454a-8763-f658bd0ce870" (UID: "cf86081d-7942-454a-8763-f658bd0ce870"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:40.877338 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.877244 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf86081d-7942-454a-8763-f658bd0ce870-tmp-dir" pod "cf86081d-7942-454a-8763-f658bd0ce870" (UID: "cf86081d-7942-454a-8763-f658bd0ce870"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:40.877473 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.877434 4440 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf86081d-7942-454a-8763-f658bd0ce870-tmp-dir\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:40.877473 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.877457 4440 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf86081d-7942-454a-8763-f658bd0ce870-home\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:40.877473 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.877470 4440 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f93d4671-a358-4547-ba77-06255ca42c3e-model-cache\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:40.877603 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.877484 4440 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf86081d-7942-454a-8763-f658bd0ce870-model-cache\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:40.877603 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.877497 4440 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f93d4671-a358-4547-ba77-06255ca42c3e-home\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:40.877603 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.877510 4440 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f93d4671-a358-4547-ba77-06255ca42c3e-tmp-dir\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:40.879322 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.879289 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf86081d-7942-454a-8763-f658bd0ce870-dshm" pod "cf86081d-7942-454a-8763-f658bd0ce870" (UID: "cf86081d-7942-454a-8763-f658bd0ce870"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:40.879552 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.879524 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f93d4671-a358-4547-ba77-06255ca42c3e-tls-certs" pod "f93d4671-a358-4547-ba77-06255ca42c3e" (UID: "f93d4671-a358-4547-ba77-06255ca42c3e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:14:40.880081 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.880044 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f93d4671-a358-4547-ba77-06255ca42c3e-kube-api-access-qk4mt" pod "f93d4671-a358-4547-ba77-06255ca42c3e" (UID: "f93d4671-a358-4547-ba77-06255ca42c3e"). InnerVolumeSpecName "kube-api-access-qk4mt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:40.880688 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.880660 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf86081d-7942-454a-8763-f658bd0ce870-kube-api-access-5d2jj" pod "cf86081d-7942-454a-8763-f658bd0ce870" (UID: "cf86081d-7942-454a-8763-f658bd0ce870"). InnerVolumeSpecName "kube-api-access-5d2jj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:40.884442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.884391 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf86081d-7942-454a-8763-f658bd0ce870-tls-certs" pod "cf86081d-7942-454a-8763-f658bd0ce870" (UID: "cf86081d-7942-454a-8763-f658bd0ce870"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:14:40.884442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.884404 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f93d4671-a358-4547-ba77-06255ca42c3e-dshm" pod "f93d4671-a358-4547-ba77-06255ca42c3e" (UID: "f93d4671-a358-4547-ba77-06255ca42c3e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:40.979147 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.979103 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d2jj\" (UniqueName: \"kubernetes.io/projected/cf86081d-7942-454a-8763-f658bd0ce870-kube-api-access-5d2jj\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:40.979147 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.979141 4440 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf86081d-7942-454a-8763-f658bd0ce870-dshm\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:40.979147 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.979154 4440 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf86081d-7942-454a-8763-f658bd0ce870-tls-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:40.979396 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.979164 4440 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f93d4671-a358-4547-ba77-06255ca42c3e-tls-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:40.979396 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.979173 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk4mt\" (UniqueName: \"kubernetes.io/projected/f93d4671-a358-4547-ba77-06255ca42c3e-kube-api-access-qk4mt\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:40.979396 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.979182 4440 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f93d4671-a358-4547-ba77-06255ca42c3e-dshm\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:40.985745 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:40.985706 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:41.400438 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:41.399356 4440 generic.go:356] "Generic (PLEG): container finished" podID="f93d4671-a358-4547-ba77-06255ca42c3e" containerID="c3ff32cc0a574168dc30068c8dcc6454aaa719461d84018cce68768fdc6b1b78" exitCode=0 Oct 31 01:14:41.400438 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:41.399525 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" event={"ID":"f93d4671-a358-4547-ba77-06255ca42c3e","Type":"ContainerDied","Data":"c3ff32cc0a574168dc30068c8dcc6454aaa719461d84018cce68768fdc6b1b78"} Oct 31 01:14:41.400438 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:41.399590 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" event={"ID":"f93d4671-a358-4547-ba77-06255ca42c3e","Type":"ContainerDied","Data":"0b497359e9a3e33043c1cd6231efda11de162feef4dd3688908af576d8ffc369"} Oct 31 01:14:41.400438 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:41.399614 4440 scope.go:122] "RemoveContainer" containerID="c3ff32cc0a574168dc30068c8dcc6454aaa719461d84018cce68768fdc6b1b78" Oct 31 01:14:41.400438 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:41.399501 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw" Oct 31 01:14:41.401590 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:41.401556 4440 generic.go:356] "Generic (PLEG): container finished" podID="cf86081d-7942-454a-8763-f658bd0ce870" containerID="c4efb30afb1028e371b427a798ca232592bde14d3173fc46f372a61d911f535f" exitCode=0 Oct 31 01:14:41.401728 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:41.401634 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" event={"ID":"cf86081d-7942-454a-8763-f658bd0ce870","Type":"ContainerDied","Data":"c4efb30afb1028e371b427a798ca232592bde14d3173fc46f372a61d911f535f"} Oct 31 01:14:41.401728 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:41.401664 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" event={"ID":"cf86081d-7942-454a-8763-f658bd0ce870","Type":"ContainerDied","Data":"5424adcadc84a2efd5dbc04457af3ec2b7e4f99ef8bda69156289a22fd99716e"} Oct 31 01:14:41.401728 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:41.401725 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878" Oct 31 01:14:41.431710 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:41.431685 4440 scope.go:122] "RemoveContainer" containerID="c3ff32cc0a574168dc30068c8dcc6454aaa719461d84018cce68768fdc6b1b78" Oct 31 01:14:41.432125 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:14:41.432103 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3ff32cc0a574168dc30068c8dcc6454aaa719461d84018cce68768fdc6b1b78\": container with ID starting with c3ff32cc0a574168dc30068c8dcc6454aaa719461d84018cce68768fdc6b1b78 not found: ID does not exist" containerID="c3ff32cc0a574168dc30068c8dcc6454aaa719461d84018cce68768fdc6b1b78" Oct 31 01:14:41.432212 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:41.432134 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3ff32cc0a574168dc30068c8dcc6454aaa719461d84018cce68768fdc6b1b78"} err="failed to get container status \"c3ff32cc0a574168dc30068c8dcc6454aaa719461d84018cce68768fdc6b1b78\": rpc error: code = NotFound desc = could not find container \"c3ff32cc0a574168dc30068c8dcc6454aaa719461d84018cce68768fdc6b1b78\": container with ID starting with c3ff32cc0a574168dc30068c8dcc6454aaa719461d84018cce68768fdc6b1b78 not found: ID does not exist" Oct 31 01:14:41.432212 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:41.432151 4440 scope.go:122] "RemoveContainer" containerID="c4efb30afb1028e371b427a798ca232592bde14d3173fc46f372a61d911f535f" Oct 31 01:14:41.447065 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:41.446994 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw"] Oct 31 01:14:41.452195 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:41.452148 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-28ff08/distinct-w5-28ff08-kserve-767868b958-zszvw"] Oct 31 01:14:41.456581 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:41.456557 4440 scope.go:122] "RemoveContainer" containerID="c4efb30afb1028e371b427a798ca232592bde14d3173fc46f372a61d911f535f" Oct 31 01:14:41.457077 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:14:41.456999 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4efb30afb1028e371b427a798ca232592bde14d3173fc46f372a61d911f535f\": container with ID starting with c4efb30afb1028e371b427a798ca232592bde14d3173fc46f372a61d911f535f not found: ID does not exist" containerID="c4efb30afb1028e371b427a798ca232592bde14d3173fc46f372a61d911f535f" Oct 31 01:14:41.457077 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:41.457037 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4efb30afb1028e371b427a798ca232592bde14d3173fc46f372a61d911f535f"} err="failed to get container status \"c4efb30afb1028e371b427a798ca232592bde14d3173fc46f372a61d911f535f\": rpc error: code = NotFound desc = could not find container \"c4efb30afb1028e371b427a798ca232592bde14d3173fc46f372a61d911f535f\": container with ID starting with c4efb30afb1028e371b427a798ca232592bde14d3173fc46f372a61d911f535f not found: ID does not exist" Oct 31 01:14:41.457569 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:41.457517 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878"] Oct 31 01:14:41.462097 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:41.462072 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-28ff08/distinct-2-w5-28ff08-kserve-7cbb68c869-wk878"] Oct 31 01:14:41.771981 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:41.771948 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf86081d-7942-454a-8763-f658bd0ce870" path="/var/lib/kubelet/pods/cf86081d-7942-454a-8763-f658bd0ce870/volumes" Oct 31 01:14:41.772470 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:41.772454 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" path="/var/lib/kubelet/pods/f93d4671-a358-4547-ba77-06255ca42c3e/volumes" Oct 31 01:14:41.778823 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:41.778794 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:42.053083 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:42.053056 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8499c8-5c65ddd74c-9r5n8" Oct 31 01:14:42.196914 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:42.196871 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab8ba016-584a-4646-a3d4-f6db4ccf0700-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab8ba016-584a-4646-a3d4-f6db4ccf0700-plugins-config-volume\") pod \"ab8ba016-584a-4646-a3d4-f6db4ccf0700\" (UID: \"ab8ba016-584a-4646-a3d4-f6db4ccf0700\") " Oct 31 01:14:42.197394 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:42.197358 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab8ba016-584a-4646-a3d4-f6db4ccf0700-plugins-config-volume" pod "ab8ba016-584a-4646-a3d4-f6db4ccf0700" (UID: "ab8ba016-584a-4646-a3d4-f6db4ccf0700"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:42.197720 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:42.197695 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab8ba016-584a-4646-a3d4-f6db4ccf0700-kube-api-access-xr2c5\" (UniqueName: \"kubernetes.io/projected/ab8ba016-584a-4646-a3d4-f6db4ccf0700-kube-api-access-xr2c5\") pod \"ab8ba016-584a-4646-a3d4-f6db4ccf0700\" (UID: \"ab8ba016-584a-4646-a3d4-f6db4ccf0700\") " Oct 31 01:14:42.198591 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:42.198562 4440 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab8ba016-584a-4646-a3d4-f6db4ccf0700-plugins-config-volume\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:42.200005 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:42.199981 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab8ba016-584a-4646-a3d4-f6db4ccf0700-kube-api-access-xr2c5" pod "ab8ba016-584a-4646-a3d4-f6db4ccf0700" (UID: "ab8ba016-584a-4646-a3d4-f6db4ccf0700"). InnerVolumeSpecName "kube-api-access-xr2c5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:42.299756 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:42.299665 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr2c5\" (UniqueName: \"kubernetes.io/projected/ab8ba016-584a-4646-a3d4-f6db4ccf0700-kube-api-access-xr2c5\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:42.412727 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:42.412689 4440 generic.go:356] "Generic (PLEG): container finished" podID="ab8ba016-584a-4646-a3d4-f6db4ccf0700" containerID="ba5565b2f2730d579c090408859c176f458b63d24c3adf03db39fe85462fe15e" exitCode=0 Oct 31 01:14:42.412929 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:42.412747 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8499c8-5c65ddd74c-9r5n8" Oct 31 01:14:42.412929 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:42.412773 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8499c8-5c65ddd74c-9r5n8" event={"ID":"ab8ba016-584a-4646-a3d4-f6db4ccf0700","Type":"ContainerDied","Data":"ba5565b2f2730d579c090408859c176f458b63d24c3adf03db39fe85462fe15e"} Oct 31 01:14:42.412929 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:42.412842 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8499c8-5c65ddd74c-9r5n8" event={"ID":"ab8ba016-584a-4646-a3d4-f6db4ccf0700","Type":"ContainerDied","Data":"fdc8c75c7337e509923238f84ca0a311a7d46943673d423c4bb2de5b393c686e"} Oct 31 01:14:42.412929 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:42.412875 4440 scope.go:122] "RemoveContainer" containerID="ba5565b2f2730d579c090408859c176f458b63d24c3adf03db39fe85462fe15e" Oct 31 01:14:42.427182 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:42.427151 4440 scope.go:122] "RemoveContainer" containerID="ba5565b2f2730d579c090408859c176f458b63d24c3adf03db39fe85462fe15e" Oct 31 01:14:42.427542 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:14:42.427518 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba5565b2f2730d579c090408859c176f458b63d24c3adf03db39fe85462fe15e\": container with ID starting with ba5565b2f2730d579c090408859c176f458b63d24c3adf03db39fe85462fe15e not found: ID does not exist" containerID="ba5565b2f2730d579c090408859c176f458b63d24c3adf03db39fe85462fe15e" Oct 31 01:14:42.427621 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:42.427550 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba5565b2f2730d579c090408859c176f458b63d24c3adf03db39fe85462fe15e"} err="failed to get container status \"ba5565b2f2730d579c090408859c176f458b63d24c3adf03db39fe85462fe15e\": rpc error: code = NotFound desc = could not find container \"ba5565b2f2730d579c090408859c176f458b63d24c3adf03db39fe85462fe15e\": container with ID starting with ba5565b2f2730d579c090408859c176f458b63d24c3adf03db39fe85462fe15e not found: ID does not exist" Oct 31 01:14:42.438157 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:42.438119 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8499c8-5c65ddd74c-9r5n8"] Oct 31 01:14:42.442500 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:42.442467 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8499c8-5c65ddd74c-9r5n8"] Oct 31 01:14:42.567296 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:42.567191 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:43.361765 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:43.361732 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:43.374565 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:43.374531 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:14:43.773491 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:43.773460 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab8ba016-584a-4646-a3d4-f6db4ccf0700" path="/var/lib/kubelet/pods/ab8ba016-584a-4646-a3d4-f6db4ccf0700/volumes" Oct 31 01:14:44.090744 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:44.090648 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:44.944395 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:44.944357 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:45.055456 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:45.054257 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:45.640319 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:45.640286 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx"] Oct 31 01:14:45.640853 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:45.640814 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" podUID="b29f36f2-9389-4f3c-91cf-658c373b901c" containerName="istio-proxy" containerID="cri-o://d077bbb89cc5bd53606420504b6da49534e3057fb292fea4d231c93ac785d681" gracePeriod=30 Oct 31 01:14:45.648800 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:45.648760 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:45.810982 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:45.810942 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:45.946456 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:45.944925 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv"] Oct 31 01:14:45.946456 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:45.945193 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" podUID="1c523fbe-6cf8-456c-acc0-463ff721c1da" containerName="istio-proxy" containerID="cri-o://a935fc080eda1214bfe404dd4704bcde9e7ac57e22580a6caef58d53be165377" gracePeriod=30 Oct 31 01:14:45.985572 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:45.985526 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf"] Oct 31 01:14:45.990739 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:45.990698 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" podUID="0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5" containerName="istio-proxy" containerID="cri-o://21e155795263856e2b406f7d54f3d4d6dcff08395039e8dc0b1f047a058c2b56" gracePeriod=30 Oct 31 01:14:46.258045 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:46.257890 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-acdaed-55478d5599-md22s"] Oct 31 01:14:46.321285 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:46.321245 4440 patch_prober.go:29] interesting pod/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:14:46.321537 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:46.321324 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" podUID="b29f36f2-9389-4f3c-91cf-658c373b901c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:14:46.351070 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:46.351025 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:46.474615 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:46.474581 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:46.587965 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:46.587859 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:46.730821 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:46.730784 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:46.763689 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:46.763621 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-7f6d1e-5754ff484f-dzhmh"] Oct 31 01:14:46.783502 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:46.783452 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7f6d1e-5754ff484f-dzhmh" podUID="f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60" containerName="maas-api" containerID="cri-o://a90d9f358e7f3f9de81333b9930a17651b2baabea3bfed1f37140f8a19e6e2db" gracePeriod=30 Oct 31 01:14:46.848075 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:46.847982 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:47.365634 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:47.365552 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7f6d1e-5754ff484f-dzhmh" Oct 31 01:14:47.383021 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:47.382968 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-28ff08-8947d86b5-frcjf" podUID="b8df1e49-cfaf-4c8c-9498-3400501ae86b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:14:47.407116 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:47.407082 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:47.473160 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:47.473125 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60-openshift-service-ca\") pod \"f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60\" (UID: \"f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60\") " Oct 31 01:14:47.473377 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:47.473251 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60-kube-api-access-94fzl\" (UniqueName: \"kubernetes.io/projected/f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60-kube-api-access-94fzl\") pod \"f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60\" (UID: \"f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60\") " Oct 31 01:14:47.473377 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:47.473357 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60-maas-api-tls\") pod \"f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60\" (UID: \"f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60\") " Oct 31 01:14:47.473555 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:47.473540 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60-metrics-certs\") pod \"f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60\" (UID: \"f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60\") " Oct 31 01:14:47.473612 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:47.473557 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60-openshift-service-ca" pod "f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60" (UID: "f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:47.474111 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:47.474091 4440 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60-openshift-service-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:47.475721 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:47.475685 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60-kube-api-access-94fzl" pod "f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60" (UID: "f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60"). InnerVolumeSpecName "kube-api-access-94fzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:47.475821 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:47.475725 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60-metrics-certs" pod "f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60" (UID: "f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:14:47.475941 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:47.475914 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60-maas-api-tls" pod "f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60" (UID: "f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:14:47.568320 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:47.568285 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:47.575583 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:47.575551 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94fzl\" (UniqueName: \"kubernetes.io/projected/f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60-kube-api-access-94fzl\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:47.575583 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:47.575582 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60-maas-api-tls\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:47.575583 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:47.575591 4440 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60-metrics-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:47.687698 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:47.687656 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:47.783667 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:47.783632 4440 generic.go:356] "Generic (PLEG): container finished" podID="f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60" containerID="a90d9f358e7f3f9de81333b9930a17651b2baabea3bfed1f37140f8a19e6e2db" exitCode=0 Oct 31 01:14:47.784192 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:47.784166 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7f6d1e-5754ff484f-dzhmh" Oct 31 01:14:47.811461 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:47.811243 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7f6d1e-5754ff484f-dzhmh" event={"ID":"f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60","Type":"ContainerDied","Data":"a90d9f358e7f3f9de81333b9930a17651b2baabea3bfed1f37140f8a19e6e2db"} Oct 31 01:14:47.811461 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:47.811317 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7f6d1e-5754ff484f-dzhmh" event={"ID":"f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60","Type":"ContainerDied","Data":"69a42b964f73585ef31eeed5cbf3c554703c4ba57b33c8c88efe14bea872c759"} Oct 31 01:14:47.811461 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:47.811348 4440 scope.go:122] "RemoveContainer" containerID="a90d9f358e7f3f9de81333b9930a17651b2baabea3bfed1f37140f8a19e6e2db" Oct 31 01:14:47.846199 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:47.846167 4440 scope.go:122] "RemoveContainer" containerID="a90d9f358e7f3f9de81333b9930a17651b2baabea3bfed1f37140f8a19e6e2db" Oct 31 01:14:47.846609 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:14:47.846582 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a90d9f358e7f3f9de81333b9930a17651b2baabea3bfed1f37140f8a19e6e2db\": container with ID starting with a90d9f358e7f3f9de81333b9930a17651b2baabea3bfed1f37140f8a19e6e2db not found: ID does not exist" containerID="a90d9f358e7f3f9de81333b9930a17651b2baabea3bfed1f37140f8a19e6e2db" Oct 31 01:14:47.846702 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:47.846620 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a90d9f358e7f3f9de81333b9930a17651b2baabea3bfed1f37140f8a19e6e2db"} err="failed to get container status \"a90d9f358e7f3f9de81333b9930a17651b2baabea3bfed1f37140f8a19e6e2db\": rpc error: code = NotFound desc = could not find container \"a90d9f358e7f3f9de81333b9930a17651b2baabea3bfed1f37140f8a19e6e2db\": container with ID starting with a90d9f358e7f3f9de81333b9930a17651b2baabea3bfed1f37140f8a19e6e2db not found: ID does not exist" Oct 31 01:14:47.850780 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:47.850747 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-7f6d1e-5754ff484f-dzhmh"] Oct 31 01:14:47.857638 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:47.857599 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-7f6d1e-5754ff484f-dzhmh"] Oct 31 01:14:48.157985 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:48.157882 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:48.281260 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:48.281227 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:48.319510 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:48.319475 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-28ff08-82b9g"] Oct 31 01:14:48.323145 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:48.323112 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-28ff08-82b9g"] Oct 31 01:14:48.400116 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:48.400081 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:48.537102 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:48.537020 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:48.920076 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:48.920041 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:49.411727 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:49.411631 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:49.752254 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:49.752218 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:49.771982 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:49.771941 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b26d9f70-bea6-42e0-b5d2-c1b394501ddf" path="/var/lib/kubelet/pods/b26d9f70-bea6-42e0-b5d2-c1b394501ddf/volumes" Oct 31 01:14:49.772559 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:49.772543 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60" path="/var/lib/kubelet/pods/f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60/volumes" Oct 31 01:14:49.882011 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:49.881977 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:50.196475 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:50.196393 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:51.259019 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.258932 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-acdaed-55478d5599-md22s" podUID="e4160358-c655-4a6b-bb39-fc8677804241" containerName="payload-pre-processing" containerID="cri-o://30fcda6740c24446c9a353da5eaecabbf6486015e8b17082862fbd1e1421fca2" gracePeriod=25 Oct 31 01:14:51.276821 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.276784 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" Oct 31 01:14:51.338520 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.336888 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c523fbe-6cf8-456c-acc0-463ff721c1da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c523fbe-6cf8-456c-acc0-463ff721c1da-credential-socket\") pod \"1c523fbe-6cf8-456c-acc0-463ff721c1da\" (UID: \"1c523fbe-6cf8-456c-acc0-463ff721c1da\") " Oct 31 01:14:51.338520 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.336953 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c523fbe-6cf8-456c-acc0-463ff721c1da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c523fbe-6cf8-456c-acc0-463ff721c1da-istio-envoy\") pod \"1c523fbe-6cf8-456c-acc0-463ff721c1da\" (UID: \"1c523fbe-6cf8-456c-acc0-463ff721c1da\") " Oct 31 01:14:51.338520 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.337003 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c523fbe-6cf8-456c-acc0-463ff721c1da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c523fbe-6cf8-456c-acc0-463ff721c1da-istiod-ca-cert\") pod \"1c523fbe-6cf8-456c-acc0-463ff721c1da\" (UID: \"1c523fbe-6cf8-456c-acc0-463ff721c1da\") " Oct 31 01:14:51.338520 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.337040 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c523fbe-6cf8-456c-acc0-463ff721c1da-kube-api-access-mvvl6\" (UniqueName: \"kubernetes.io/projected/1c523fbe-6cf8-456c-acc0-463ff721c1da-kube-api-access-mvvl6\") pod \"1c523fbe-6cf8-456c-acc0-463ff721c1da\" (UID: \"1c523fbe-6cf8-456c-acc0-463ff721c1da\") " Oct 31 01:14:51.338520 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.337120 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c523fbe-6cf8-456c-acc0-463ff721c1da-istio-token\" (UniqueName: \"kubernetes.io/projected/1c523fbe-6cf8-456c-acc0-463ff721c1da-istio-token\") pod \"1c523fbe-6cf8-456c-acc0-463ff721c1da\" (UID: \"1c523fbe-6cf8-456c-acc0-463ff721c1da\") " Oct 31 01:14:51.338520 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.337194 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c523fbe-6cf8-456c-acc0-463ff721c1da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c523fbe-6cf8-456c-acc0-463ff721c1da-workload-socket\") pod \"1c523fbe-6cf8-456c-acc0-463ff721c1da\" (UID: \"1c523fbe-6cf8-456c-acc0-463ff721c1da\") " Oct 31 01:14:51.338520 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.337307 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c523fbe-6cf8-456c-acc0-463ff721c1da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c523fbe-6cf8-456c-acc0-463ff721c1da-workload-certs\") pod \"1c523fbe-6cf8-456c-acc0-463ff721c1da\" (UID: \"1c523fbe-6cf8-456c-acc0-463ff721c1da\") " Oct 31 01:14:51.338520 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.337350 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c523fbe-6cf8-456c-acc0-463ff721c1da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c523fbe-6cf8-456c-acc0-463ff721c1da-istio-data\") pod \"1c523fbe-6cf8-456c-acc0-463ff721c1da\" (UID: \"1c523fbe-6cf8-456c-acc0-463ff721c1da\") " Oct 31 01:14:51.338520 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.337453 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c523fbe-6cf8-456c-acc0-463ff721c1da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c523fbe-6cf8-456c-acc0-463ff721c1da-istio-podinfo\") pod \"1c523fbe-6cf8-456c-acc0-463ff721c1da\" (UID: \"1c523fbe-6cf8-456c-acc0-463ff721c1da\") " Oct 31 01:14:51.341115 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.340593 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c523fbe-6cf8-456c-acc0-463ff721c1da-workload-certs" pod "1c523fbe-6cf8-456c-acc0-463ff721c1da" (UID: "1c523fbe-6cf8-456c-acc0-463ff721c1da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:51.341115 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.340616 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c523fbe-6cf8-456c-acc0-463ff721c1da-istio-data" pod "1c523fbe-6cf8-456c-acc0-463ff721c1da" (UID: "1c523fbe-6cf8-456c-acc0-463ff721c1da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:51.341115 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.340704 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c523fbe-6cf8-456c-acc0-463ff721c1da-credential-socket" pod "1c523fbe-6cf8-456c-acc0-463ff721c1da" (UID: "1c523fbe-6cf8-456c-acc0-463ff721c1da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:51.341115 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.341086 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c523fbe-6cf8-456c-acc0-463ff721c1da-istiod-ca-cert" pod "1c523fbe-6cf8-456c-acc0-463ff721c1da" (UID: "1c523fbe-6cf8-456c-acc0-463ff721c1da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:51.342105 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.342073 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c523fbe-6cf8-456c-acc0-463ff721c1da-workload-socket" pod "1c523fbe-6cf8-456c-acc0-463ff721c1da" (UID: "1c523fbe-6cf8-456c-acc0-463ff721c1da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:51.347506 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.345566 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c523fbe-6cf8-456c-acc0-463ff721c1da-kube-api-access-mvvl6" pod "1c523fbe-6cf8-456c-acc0-463ff721c1da" (UID: "1c523fbe-6cf8-456c-acc0-463ff721c1da"). InnerVolumeSpecName "kube-api-access-mvvl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:51.347820 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.347767 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c523fbe-6cf8-456c-acc0-463ff721c1da-istio-envoy" pod "1c523fbe-6cf8-456c-acc0-463ff721c1da" (UID: "1c523fbe-6cf8-456c-acc0-463ff721c1da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:51.348458 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.348387 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c523fbe-6cf8-456c-acc0-463ff721c1da-istio-token" pod "1c523fbe-6cf8-456c-acc0-463ff721c1da" (UID: "1c523fbe-6cf8-456c-acc0-463ff721c1da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:51.348658 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.348631 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c523fbe-6cf8-456c-acc0-463ff721c1da-istio-podinfo" pod "1c523fbe-6cf8-456c-acc0-463ff721c1da" (UID: "1c523fbe-6cf8-456c-acc0-463ff721c1da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:14:51.439086 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.439051 4440 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c523fbe-6cf8-456c-acc0-463ff721c1da-workload-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:51.439086 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.439086 4440 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c523fbe-6cf8-456c-acc0-463ff721c1da-istio-data\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:51.439303 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.439130 4440 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c523fbe-6cf8-456c-acc0-463ff721c1da-istio-podinfo\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:51.439303 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.439144 4440 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c523fbe-6cf8-456c-acc0-463ff721c1da-credential-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:51.439303 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.439160 4440 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c523fbe-6cf8-456c-acc0-463ff721c1da-istio-envoy\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:51.439303 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.439173 4440 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c523fbe-6cf8-456c-acc0-463ff721c1da-istiod-ca-cert\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:51.439303 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.439187 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvvl6\" (UniqueName: \"kubernetes.io/projected/1c523fbe-6cf8-456c-acc0-463ff721c1da-kube-api-access-mvvl6\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:51.439303 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.439201 4440 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c523fbe-6cf8-456c-acc0-463ff721c1da-istio-token\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:51.439303 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.439213 4440 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c523fbe-6cf8-456c-acc0-463ff721c1da-workload-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:51.441576 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.441552 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" Oct 31 01:14:51.540286 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.540181 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-istio-token\" (UniqueName: \"kubernetes.io/projected/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-istio-token\") pod \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\" (UID: \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\") " Oct 31 01:14:51.540286 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.540242 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-workload-certs\") pod \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\" (UID: \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\") " Oct 31 01:14:51.540546 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.540297 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-workload-socket\") pod \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\" (UID: \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\") " Oct 31 01:14:51.540546 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.540478 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-istiod-ca-cert\") pod \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\" (UID: \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\") " Oct 31 01:14:51.540546 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.540541 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-istio-envoy\") pod \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\" (UID: \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\") " Oct 31 01:14:51.540710 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.540586 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-istio-podinfo\") pod \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\" (UID: \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\") " Oct 31 01:14:51.540710 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.540695 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-istio-data\") pod \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\" (UID: \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\") " Oct 31 01:14:51.540820 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.540760 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-credential-socket\") pod \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\" (UID: \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\") " Oct 31 01:14:51.540820 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.540784 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-kube-api-access-8xfpx\" (UniqueName: \"kubernetes.io/projected/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-kube-api-access-8xfpx\") pod \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\" (UID: \"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5\") " Oct 31 01:14:51.541792 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.541759 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-workload-certs" pod "0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5" (UID: "0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:51.542941 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.542884 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-istiod-ca-cert" pod "0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5" (UID: "0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:51.543095 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.542946 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-istio-data" pod "0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5" (UID: "0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:51.543095 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.543020 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-credential-socket" pod "0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5" (UID: "0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:51.543259 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.543238 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-workload-socket" pod "0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5" (UID: "0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:51.545326 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.545284 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-istio-envoy" pod "0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5" (UID: "0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:51.545601 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.545570 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-istio-token" pod "0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5" (UID: "0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:51.546203 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.546180 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-kube-api-access-8xfpx" pod "0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5" (UID: "0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5"). InnerVolumeSpecName "kube-api-access-8xfpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:51.547222 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.547190 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-istio-podinfo" pod "0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5" (UID: "0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:14:51.656166 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.655963 4440 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-workload-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:51.656166 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.656006 4440 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-istiod-ca-cert\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:51.656166 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.656021 4440 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-istio-envoy\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:51.656166 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.656035 4440 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-istio-podinfo\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:51.656166 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.656049 4440 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-istio-data\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:51.656166 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.656062 4440 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-credential-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:51.656166 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.656075 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xfpx\" (UniqueName: \"kubernetes.io/projected/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-kube-api-access-8xfpx\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:51.656166 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.656090 4440 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-istio-token\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:51.656166 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.656102 4440 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5-workload-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:51.824497 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.824352 4440 generic.go:356] "Generic (PLEG): container finished" podID="1c523fbe-6cf8-456c-acc0-463ff721c1da" containerID="a935fc080eda1214bfe404dd4704bcde9e7ac57e22580a6caef58d53be165377" exitCode=0 Oct 31 01:14:51.824497 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.824402 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" Oct 31 01:14:51.824497 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.824469 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" event={"ID":"1c523fbe-6cf8-456c-acc0-463ff721c1da","Type":"ContainerDied","Data":"a935fc080eda1214bfe404dd4704bcde9e7ac57e22580a6caef58d53be165377"} Oct 31 01:14:51.824765 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.824540 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv" event={"ID":"1c523fbe-6cf8-456c-acc0-463ff721c1da","Type":"ContainerDied","Data":"7dd11996c63e3dc5ea1e1036d5209b834bd298c5c5129c12b6c36e2d96b40d02"} Oct 31 01:14:51.824765 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.824572 4440 scope.go:122] "RemoveContainer" containerID="a935fc080eda1214bfe404dd4704bcde9e7ac57e22580a6caef58d53be165377" Oct 31 01:14:51.826739 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.826709 4440 generic.go:356] "Generic (PLEG): container finished" podID="0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5" containerID="21e155795263856e2b406f7d54f3d4d6dcff08395039e8dc0b1f047a058c2b56" exitCode=0 Oct 31 01:14:51.826936 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.826877 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" Oct 31 01:14:51.827626 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.827517 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" event={"ID":"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5","Type":"ContainerDied","Data":"21e155795263856e2b406f7d54f3d4d6dcff08395039e8dc0b1f047a058c2b56"} Oct 31 01:14:51.827626 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.827581 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf" event={"ID":"0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5","Type":"ContainerDied","Data":"d4cb61f9f229aeff75897b408c7fd4ee9dbdb8a2affa9bfef64e19c426519c49"} Oct 31 01:14:51.830920 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.830893 4440 generic.go:356] "Generic (PLEG): container finished" podID="b29f36f2-9389-4f3c-91cf-658c373b901c" containerID="d077bbb89cc5bd53606420504b6da49534e3057fb292fea4d231c93ac785d681" exitCode=0 Oct 31 01:14:51.831045 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.830961 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" event={"ID":"b29f36f2-9389-4f3c-91cf-658c373b901c","Type":"ContainerDied","Data":"d077bbb89cc5bd53606420504b6da49534e3057fb292fea4d231c93ac785d681"} Oct 31 01:14:51.837948 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.837926 4440 scope.go:122] "RemoveContainer" containerID="a935fc080eda1214bfe404dd4704bcde9e7ac57e22580a6caef58d53be165377" Oct 31 01:14:51.838253 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:14:51.838230 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a935fc080eda1214bfe404dd4704bcde9e7ac57e22580a6caef58d53be165377\": container with ID starting with a935fc080eda1214bfe404dd4704bcde9e7ac57e22580a6caef58d53be165377 not found: ID does not exist" containerID="a935fc080eda1214bfe404dd4704bcde9e7ac57e22580a6caef58d53be165377" Oct 31 01:14:51.838318 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.838265 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a935fc080eda1214bfe404dd4704bcde9e7ac57e22580a6caef58d53be165377"} err="failed to get container status \"a935fc080eda1214bfe404dd4704bcde9e7ac57e22580a6caef58d53be165377\": rpc error: code = NotFound desc = could not find container \"a935fc080eda1214bfe404dd4704bcde9e7ac57e22580a6caef58d53be165377\": container with ID starting with a935fc080eda1214bfe404dd4704bcde9e7ac57e22580a6caef58d53be165377 not found: ID does not exist" Oct 31 01:14:51.838318 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.838286 4440 scope.go:122] "RemoveContainer" containerID="21e155795263856e2b406f7d54f3d4d6dcff08395039e8dc0b1f047a058c2b56" Oct 31 01:14:51.848279 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.848250 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf"] Oct 31 01:14:51.851389 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.851366 4440 scope.go:122] "RemoveContainer" containerID="21e155795263856e2b406f7d54f3d4d6dcff08395039e8dc0b1f047a058c2b56" Oct 31 01:14:51.851738 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:14:51.851718 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21e155795263856e2b406f7d54f3d4d6dcff08395039e8dc0b1f047a058c2b56\": container with ID starting with 21e155795263856e2b406f7d54f3d4d6dcff08395039e8dc0b1f047a058c2b56 not found: ID does not exist" containerID="21e155795263856e2b406f7d54f3d4d6dcff08395039e8dc0b1f047a058c2b56" Oct 31 01:14:51.851814 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.851745 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21e155795263856e2b406f7d54f3d4d6dcff08395039e8dc0b1f047a058c2b56"} err="failed to get container status \"21e155795263856e2b406f7d54f3d4d6dcff08395039e8dc0b1f047a058c2b56\": rpc error: code = NotFound desc = could not find container \"21e155795263856e2b406f7d54f3d4d6dcff08395039e8dc0b1f047a058c2b56\": container with ID starting with 21e155795263856e2b406f7d54f3d4d6dcff08395039e8dc0b1f047a058c2b56 not found: ID does not exist" Oct 31 01:14:51.852364 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.852339 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kqzsf"] Oct 31 01:14:51.865495 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.865468 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv"] Oct 31 01:14:51.869017 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.868992 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-ksmcv"] Oct 31 01:14:51.948133 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:51.948037 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:52.084012 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:52.083932 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" Oct 31 01:14:52.163783 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:52.163744 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b29f36f2-9389-4f3c-91cf-658c373b901c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b29f36f2-9389-4f3c-91cf-658c373b901c-istio-podinfo\") pod \"b29f36f2-9389-4f3c-91cf-658c373b901c\" (UID: \"b29f36f2-9389-4f3c-91cf-658c373b901c\") " Oct 31 01:14:52.163986 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:52.163814 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b29f36f2-9389-4f3c-91cf-658c373b901c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b29f36f2-9389-4f3c-91cf-658c373b901c-istio-data\") pod \"b29f36f2-9389-4f3c-91cf-658c373b901c\" (UID: \"b29f36f2-9389-4f3c-91cf-658c373b901c\") " Oct 31 01:14:52.163986 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:52.163850 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b29f36f2-9389-4f3c-91cf-658c373b901c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b29f36f2-9389-4f3c-91cf-658c373b901c-istio-envoy\") pod \"b29f36f2-9389-4f3c-91cf-658c373b901c\" (UID: \"b29f36f2-9389-4f3c-91cf-658c373b901c\") " Oct 31 01:14:52.163986 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:52.163931 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b29f36f2-9389-4f3c-91cf-658c373b901c-istio-token\" (UniqueName: \"kubernetes.io/projected/b29f36f2-9389-4f3c-91cf-658c373b901c-istio-token\") pod \"b29f36f2-9389-4f3c-91cf-658c373b901c\" (UID: \"b29f36f2-9389-4f3c-91cf-658c373b901c\") " Oct 31 01:14:52.163986 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:52.163963 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b29f36f2-9389-4f3c-91cf-658c373b901c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b29f36f2-9389-4f3c-91cf-658c373b901c-workload-socket\") pod \"b29f36f2-9389-4f3c-91cf-658c373b901c\" (UID: \"b29f36f2-9389-4f3c-91cf-658c373b901c\") " Oct 31 01:14:52.164201 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:52.164045 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b29f36f2-9389-4f3c-91cf-658c373b901c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b29f36f2-9389-4f3c-91cf-658c373b901c-credential-socket\") pod \"b29f36f2-9389-4f3c-91cf-658c373b901c\" (UID: \"b29f36f2-9389-4f3c-91cf-658c373b901c\") " Oct 31 01:14:52.164201 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:52.164134 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b29f36f2-9389-4f3c-91cf-658c373b901c-kube-api-access-m5w4l\" (UniqueName: \"kubernetes.io/projected/b29f36f2-9389-4f3c-91cf-658c373b901c-kube-api-access-m5w4l\") pod \"b29f36f2-9389-4f3c-91cf-658c373b901c\" (UID: \"b29f36f2-9389-4f3c-91cf-658c373b901c\") " Oct 31 01:14:52.164201 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:52.164167 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b29f36f2-9389-4f3c-91cf-658c373b901c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b29f36f2-9389-4f3c-91cf-658c373b901c-istiod-ca-cert\") pod \"b29f36f2-9389-4f3c-91cf-658c373b901c\" (UID: \"b29f36f2-9389-4f3c-91cf-658c373b901c\") " Oct 31 01:14:52.164201 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:52.164178 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b29f36f2-9389-4f3c-91cf-658c373b901c-istio-data" pod "b29f36f2-9389-4f3c-91cf-658c373b901c" (UID: "b29f36f2-9389-4f3c-91cf-658c373b901c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:52.164404 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:52.164197 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b29f36f2-9389-4f3c-91cf-658c373b901c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b29f36f2-9389-4f3c-91cf-658c373b901c-workload-certs\") pod \"b29f36f2-9389-4f3c-91cf-658c373b901c\" (UID: \"b29f36f2-9389-4f3c-91cf-658c373b901c\") " Oct 31 01:14:52.164404 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:52.164326 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b29f36f2-9389-4f3c-91cf-658c373b901c-workload-socket" pod "b29f36f2-9389-4f3c-91cf-658c373b901c" (UID: "b29f36f2-9389-4f3c-91cf-658c373b901c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:52.164404 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:52.164364 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b29f36f2-9389-4f3c-91cf-658c373b901c-credential-socket" pod "b29f36f2-9389-4f3c-91cf-658c373b901c" (UID: "b29f36f2-9389-4f3c-91cf-658c373b901c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:52.164689 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:52.164665 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b29f36f2-9389-4f3c-91cf-658c373b901c-workload-certs" pod "b29f36f2-9389-4f3c-91cf-658c373b901c" (UID: "b29f36f2-9389-4f3c-91cf-658c373b901c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:52.164783 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:52.164724 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b29f36f2-9389-4f3c-91cf-658c373b901c-istiod-ca-cert" pod "b29f36f2-9389-4f3c-91cf-658c373b901c" (UID: "b29f36f2-9389-4f3c-91cf-658c373b901c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:52.164934 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:52.164917 4440 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b29f36f2-9389-4f3c-91cf-658c373b901c-workload-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:52.164993 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:52.164934 4440 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b29f36f2-9389-4f3c-91cf-658c373b901c-credential-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:52.164993 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:52.164943 4440 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b29f36f2-9389-4f3c-91cf-658c373b901c-istiod-ca-cert\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:52.164993 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:52.164952 4440 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b29f36f2-9389-4f3c-91cf-658c373b901c-workload-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:52.164993 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:52.164960 4440 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b29f36f2-9389-4f3c-91cf-658c373b901c-istio-data\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:52.166221 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:52.166193 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b29f36f2-9389-4f3c-91cf-658c373b901c-istio-envoy" pod "b29f36f2-9389-4f3c-91cf-658c373b901c" (UID: "b29f36f2-9389-4f3c-91cf-658c373b901c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:14:52.166221 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:52.166214 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b29f36f2-9389-4f3c-91cf-658c373b901c-istio-podinfo" pod "b29f36f2-9389-4f3c-91cf-658c373b901c" (UID: "b29f36f2-9389-4f3c-91cf-658c373b901c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:14:52.166590 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:52.166573 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b29f36f2-9389-4f3c-91cf-658c373b901c-istio-token" pod "b29f36f2-9389-4f3c-91cf-658c373b901c" (UID: "b29f36f2-9389-4f3c-91cf-658c373b901c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:52.166646 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:52.166604 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b29f36f2-9389-4f3c-91cf-658c373b901c-kube-api-access-m5w4l" pod "b29f36f2-9389-4f3c-91cf-658c373b901c" (UID: "b29f36f2-9389-4f3c-91cf-658c373b901c"). InnerVolumeSpecName "kube-api-access-m5w4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:52.266166 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:52.266128 4440 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b29f36f2-9389-4f3c-91cf-658c373b901c-istio-podinfo\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:52.266166 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:52.266160 4440 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b29f36f2-9389-4f3c-91cf-658c373b901c-istio-envoy\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:52.266166 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:52.266171 4440 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b29f36f2-9389-4f3c-91cf-658c373b901c-istio-token\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:52.266665 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:52.266181 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5w4l\" (UniqueName: \"kubernetes.io/projected/b29f36f2-9389-4f3c-91cf-658c373b901c-kube-api-access-m5w4l\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:52.398991 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:52.398889 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:52.843235 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:52.843197 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" event={"ID":"b29f36f2-9389-4f3c-91cf-658c373b901c","Type":"ContainerDied","Data":"2fe8ddc03376c605c03ced5da69ed746eef52df0c301ea8bf2a6110addfc3ed1"} Oct 31 01:14:52.843484 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:52.843263 4440 scope.go:122] "RemoveContainer" containerID="d077bbb89cc5bd53606420504b6da49534e3057fb292fea4d231c93ac785d681" Oct 31 01:14:52.843484 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:52.843203 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx" Oct 31 01:14:52.870853 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:52.870711 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx"] Oct 31 01:14:52.874542 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:52.874506 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8499c8-openshift-default-8587968855-kg7gx"] Oct 31 01:14:53.063302 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:53.063267 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:53.669175 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:53.669134 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:53.772755 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:53.772716 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5" path="/var/lib/kubelet/pods/0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5/volumes" Oct 31 01:14:53.773323 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:53.773305 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c523fbe-6cf8-456c-acc0-463ff721c1da" path="/var/lib/kubelet/pods/1c523fbe-6cf8-456c-acc0-463ff721c1da/volumes" Oct 31 01:14:53.773790 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:53.773777 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b29f36f2-9389-4f3c-91cf-658c373b901c" path="/var/lib/kubelet/pods/b29f36f2-9389-4f3c-91cf-658c373b901c/volumes" Oct 31 01:14:54.586058 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:54.586017 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:54.787251 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:54.787210 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:54.905548 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:54.905446 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:55.077041 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:55.067791 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph"] Oct 31 01:14:55.077041 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:55.068060 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" podUID="b192b955-7018-4d48-b3c6-c58d86eeb5b2" containerName="istio-proxy" containerID="cri-o://8a419a62c34d493bacbfce57925ee0e68c8d090600939ab0e06825ba43becc17" gracePeriod=30 Oct 31 01:14:55.077041 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:55.072947 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh"] Oct 31 01:14:55.077041 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:55.073310 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" podUID="8f8c2432-9405-4f4f-b24f-1b73ea0da86a" containerName="istio-proxy" containerID="cri-o://3268df394c99ef1132a6a6b2280a81753857a6722c453d7681cb7f6e93d56da6" gracePeriod=30 Oct 31 01:14:55.093680 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:55.093644 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4"] Oct 31 01:14:55.093938 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:55.093907 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" podUID="3b4e5ddd-9933-47e1-9265-085858232375" containerName="istio-proxy" containerID="cri-o://9159b732b08f6092f3f7cf448af44053edff619b23624a91e1b5ae0f215011e0" gracePeriod=30 Oct 31 01:14:55.120529 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:55.120445 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:55.524946 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:55.524899 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:55.752133 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:55.752094 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:55.863137 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:55.863033 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:56.075537 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:56.075499 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw"] Oct 31 01:14:56.075847 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:56.075817 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" podUID="7ee3eb69-94de-447c-b9ec-194a0458f2e7" containerName="istio-proxy" containerID="cri-o://6ced7779087cd90b78a891696327fcffe5daf642c5fe82ef100d9f7576911c49" gracePeriod=30 Oct 31 01:14:56.083265 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:56.083232 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8"] Oct 31 01:14:56.083537 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:56.083512 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" podUID="81095022-acb2-4b1c-acc6-a9501b0ea478" containerName="istio-proxy" containerID="cri-o://eade63983f78f236269934cc635215faffbfcadf3990e2d1fb7b34ccf28dda43" gracePeriod=30 Oct 31 01:14:56.088445 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:56.088225 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58"] Oct 31 01:14:56.088586 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:56.088498 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" podUID="5f420497-f98d-4a0c-8552-fca5c3f002f2" containerName="istio-proxy" containerID="cri-o://43a4bbc42c552d99a20b0ed441a4ece0bbaab3bcb4f7a2cfa71b6d4f1e76af2c" gracePeriod=30 Oct 31 01:14:56.152641 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:56.151527 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:56.372252 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:56.371383 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-acdaed-55478d5599-md22s" podUID="e4160358-c655-4a6b-bb39-fc8677804241" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:14:56.395924 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:56.395894 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-28ff08-8947d86b5-frcjf" Oct 31 01:14:56.450664 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:56.449883 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8df1e49-cfaf-4c8c-9498-3400501ae86b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8df1e49-cfaf-4c8c-9498-3400501ae86b-plugins-config-volume\") pod \"b8df1e49-cfaf-4c8c-9498-3400501ae86b\" (UID: \"b8df1e49-cfaf-4c8c-9498-3400501ae86b\") " Oct 31 01:14:56.450664 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:56.450089 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8df1e49-cfaf-4c8c-9498-3400501ae86b-kube-api-access-x2bvc\" (UniqueName: \"kubernetes.io/projected/b8df1e49-cfaf-4c8c-9498-3400501ae86b-kube-api-access-x2bvc\") pod \"b8df1e49-cfaf-4c8c-9498-3400501ae86b\" (UID: \"b8df1e49-cfaf-4c8c-9498-3400501ae86b\") " Oct 31 01:14:56.450905 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:56.450668 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8df1e49-cfaf-4c8c-9498-3400501ae86b-plugins-config-volume" pod "b8df1e49-cfaf-4c8c-9498-3400501ae86b" (UID: "b8df1e49-cfaf-4c8c-9498-3400501ae86b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:56.453179 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:56.453145 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8df1e49-cfaf-4c8c-9498-3400501ae86b-kube-api-access-x2bvc" pod "b8df1e49-cfaf-4c8c-9498-3400501ae86b" (UID: "b8df1e49-cfaf-4c8c-9498-3400501ae86b"). InnerVolumeSpecName "kube-api-access-x2bvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:56.515775 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:56.515729 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:56.552957 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:56.552912 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2bvc\" (UniqueName: \"kubernetes.io/projected/b8df1e49-cfaf-4c8c-9498-3400501ae86b-kube-api-access-x2bvc\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:56.552957 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:56.552958 4440 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8df1e49-cfaf-4c8c-9498-3400501ae86b-plugins-config-volume\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:56.718524 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:56.718486 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:56.878453 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:56.878383 4440 generic.go:356] "Generic (PLEG): container finished" podID="b8df1e49-cfaf-4c8c-9498-3400501ae86b" containerID="132e471216ed1fe2da347dc8ccaf1379ba6bac80ff6fd23338cbd50509a26a0c" exitCode=0 Oct 31 01:14:56.878927 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:56.878454 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-28ff08-8947d86b5-frcjf" event={"ID":"b8df1e49-cfaf-4c8c-9498-3400501ae86b","Type":"ContainerDied","Data":"132e471216ed1fe2da347dc8ccaf1379ba6bac80ff6fd23338cbd50509a26a0c"} Oct 31 01:14:56.878927 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:56.878522 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-28ff08-8947d86b5-frcjf" event={"ID":"b8df1e49-cfaf-4c8c-9498-3400501ae86b","Type":"ContainerDied","Data":"fcb90366b81fcd3c25060076a4a89225f9444e67bffa4ebb0934a2971844e628"} Oct 31 01:14:56.878927 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:56.878465 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-28ff08-8947d86b5-frcjf" Oct 31 01:14:56.878927 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:56.878557 4440 scope.go:122] "RemoveContainer" containerID="132e471216ed1fe2da347dc8ccaf1379ba6bac80ff6fd23338cbd50509a26a0c" Oct 31 01:14:56.892411 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:56.892388 4440 scope.go:122] "RemoveContainer" containerID="132e471216ed1fe2da347dc8ccaf1379ba6bac80ff6fd23338cbd50509a26a0c" Oct 31 01:14:56.893510 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:14:56.893482 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"132e471216ed1fe2da347dc8ccaf1379ba6bac80ff6fd23338cbd50509a26a0c\": container with ID starting with 132e471216ed1fe2da347dc8ccaf1379ba6bac80ff6fd23338cbd50509a26a0c not found: ID does not exist" containerID="132e471216ed1fe2da347dc8ccaf1379ba6bac80ff6fd23338cbd50509a26a0c" Oct 31 01:14:56.893623 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:56.893522 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"132e471216ed1fe2da347dc8ccaf1379ba6bac80ff6fd23338cbd50509a26a0c"} err="failed to get container status \"132e471216ed1fe2da347dc8ccaf1379ba6bac80ff6fd23338cbd50509a26a0c\": rpc error: code = NotFound desc = could not find container \"132e471216ed1fe2da347dc8ccaf1379ba6bac80ff6fd23338cbd50509a26a0c\": container with ID starting with 132e471216ed1fe2da347dc8ccaf1379ba6bac80ff6fd23338cbd50509a26a0c not found: ID does not exist" Oct 31 01:14:56.917215 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:56.917172 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-28ff08-8947d86b5-frcjf"] Oct 31 01:14:56.919757 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:56.919729 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-28ff08-8947d86b5-frcjf"] Oct 31 01:14:57.181264 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:57.181227 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:57.772598 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:57.772560 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8df1e49-cfaf-4c8c-9498-3400501ae86b" path="/var/lib/kubelet/pods/b8df1e49-cfaf-4c8c-9498-3400501ae86b/volumes" Oct 31 01:14:58.010957 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:58.010921 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:58.478041 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:58.478007 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:58.843543 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:58.843409 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:59.412352 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:59.412310 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-acdaed-7ccf869bd7-ldg5t"] Oct 31 01:14:59.412846 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:59.412584 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-acdaed-7ccf869bd7-ldg5t" podUID="0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3" containerName="maas-api" containerID="cri-o://7b9979ea3a8417f6f75a18b6991117c96c64ce869fee9633f9506a84ca22bf47" gracePeriod=30 Oct 31 01:14:59.441592 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:59.441552 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:59.608059 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:59.607847 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:59.743219 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:59.743073 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:59.824543 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:59.824509 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-acdaed-7ccf869bd7-ldg5t" Oct 31 01:14:59.860465 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:59.860408 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3-openshift-service-ca\") pod \"0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3\" (UID: \"0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3\") " Oct 31 01:14:59.860669 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:59.860473 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3-maas-api-tls\") pod \"0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3\" (UID: \"0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3\") " Oct 31 01:14:59.860669 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:59.860606 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3-kube-api-access-592cg\" (UniqueName: \"kubernetes.io/projected/0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3-kube-api-access-592cg\") pod \"0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3\" (UID: \"0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3\") " Oct 31 01:14:59.860669 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:59.860654 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3-metrics-certs\") pod \"0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3\" (UID: \"0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3\") " Oct 31 01:14:59.860853 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:59.860826 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3-openshift-service-ca" pod "0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3" (UID: "0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:59.861261 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:59.861238 4440 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3-openshift-service-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:59.864686 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:59.864655 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3-maas-api-tls" pod "0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3" (UID: "0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:14:59.864815 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:59.864683 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3-metrics-certs" pod "0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3" (UID: "0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:14:59.865289 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:59.865267 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3-kube-api-access-592cg" pod "0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3" (UID: "0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3"). InnerVolumeSpecName "kube-api-access-592cg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:59.872540 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:59.872505 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:59.919925 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:59.919886 4440 generic.go:356] "Generic (PLEG): container finished" podID="0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3" containerID="7b9979ea3a8417f6f75a18b6991117c96c64ce869fee9633f9506a84ca22bf47" exitCode=0 Oct 31 01:14:59.920107 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:59.919964 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-acdaed-7ccf869bd7-ldg5t" Oct 31 01:14:59.920107 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:59.919969 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-acdaed-7ccf869bd7-ldg5t" event={"ID":"0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3","Type":"ContainerDied","Data":"7b9979ea3a8417f6f75a18b6991117c96c64ce869fee9633f9506a84ca22bf47"} Oct 31 01:14:59.920107 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:59.920072 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-acdaed-7ccf869bd7-ldg5t" event={"ID":"0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3","Type":"ContainerDied","Data":"f570bdf9e1c1963f218b087582ed41d70478dd859634cedd5e35e98a231c9e1c"} Oct 31 01:14:59.920107 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:59.920105 4440 scope.go:122] "RemoveContainer" containerID="7b9979ea3a8417f6f75a18b6991117c96c64ce869fee9633f9506a84ca22bf47" Oct 31 01:14:59.931824 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:59.931736 4440 scope.go:122] "RemoveContainer" containerID="7b9979ea3a8417f6f75a18b6991117c96c64ce869fee9633f9506a84ca22bf47" Oct 31 01:14:59.936313 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:14:59.936277 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b9979ea3a8417f6f75a18b6991117c96c64ce869fee9633f9506a84ca22bf47\": container with ID starting with 7b9979ea3a8417f6f75a18b6991117c96c64ce869fee9633f9506a84ca22bf47 not found: ID does not exist" containerID="7b9979ea3a8417f6f75a18b6991117c96c64ce869fee9633f9506a84ca22bf47" Oct 31 01:14:59.936475 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:59.936324 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b9979ea3a8417f6f75a18b6991117c96c64ce869fee9633f9506a84ca22bf47"} err="failed to get container status \"7b9979ea3a8417f6f75a18b6991117c96c64ce869fee9633f9506a84ca22bf47\": rpc error: code = NotFound desc = could not find container \"7b9979ea3a8417f6f75a18b6991117c96c64ce869fee9633f9506a84ca22bf47\": container with ID starting with 7b9979ea3a8417f6f75a18b6991117c96c64ce869fee9633f9506a84ca22bf47 not found: ID does not exist" Oct 31 01:14:59.958494 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:59.958454 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-acdaed-7ccf869bd7-ldg5t"] Oct 31 01:14:59.964466 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:59.963008 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-592cg\" (UniqueName: \"kubernetes.io/projected/0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3-kube-api-access-592cg\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:59.964466 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:59.963069 4440 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3-metrics-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:59.964466 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:59.963087 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3-maas-api-tls\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:14:59.964466 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:14:59.963701 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-acdaed-7ccf869bd7-ldg5t"] Oct 31 01:15:00.147224 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.147185 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-487a6c-29890155-gpd4t"] Oct 31 01:15:00.148046 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148023 4440 cpu_manager.go:414] "containerMap: removing container" podUID="dc34b9f5-85b1-4cab-a49e-c2106d685cc2" containerName="istio-proxy" Oct 31 01:15:00.148114 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148050 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc34b9f5-85b1-4cab-a49e-c2106d685cc2" containerName="istio-proxy" Oct 31 01:15:00.148114 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148069 4440 cpu_manager.go:414] "containerMap: removing container" podUID="0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3" containerName="maas-api" Oct 31 01:15:00.148114 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148077 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3" containerName="maas-api" Oct 31 01:15:00.148114 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148102 4440 cpu_manager.go:414] "containerMap: removing container" podUID="7f77962d-e14f-4225-b2e9-7a709dc8c4f4" containerName="revoke-keys" Oct 31 01:15:00.148114 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148109 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f77962d-e14f-4225-b2e9-7a709dc8c4f4" containerName="revoke-keys" Oct 31 01:15:00.148256 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148124 4440 cpu_manager.go:414] "containerMap: removing container" podUID="583a1183-7c84-40ce-aecb-9663f82a8a47" containerName="istio-proxy" Oct 31 01:15:00.148256 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148132 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="583a1183-7c84-40ce-aecb-9663f82a8a47" containerName="istio-proxy" Oct 31 01:15:00.148256 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148145 4440 cpu_manager.go:414] "containerMap: removing container" podUID="eeb7394c-0c4a-43e6-bbca-f4222bec42ff" containerName="istio-proxy" Oct 31 01:15:00.148256 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148152 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eeb7394c-0c4a-43e6-bbca-f4222bec42ff" containerName="istio-proxy" Oct 31 01:15:00.148256 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148172 4440 cpu_manager.go:414] "containerMap: removing container" podUID="eaad3e7d-1293-43d5-9fe6-674e69ea840e" containerName="payload-pre-processing" Oct 31 01:15:00.148256 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148180 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaad3e7d-1293-43d5-9fe6-674e69ea840e" containerName="payload-pre-processing" Oct 31 01:15:00.148256 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148192 4440 cpu_manager.go:414] "containerMap: removing container" podUID="0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5" containerName="istio-proxy" Oct 31 01:15:00.148256 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148199 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5" containerName="istio-proxy" Oct 31 01:15:00.148256 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148214 4440 cpu_manager.go:414] "containerMap: removing container" podUID="1c523fbe-6cf8-456c-acc0-463ff721c1da" containerName="istio-proxy" Oct 31 01:15:00.148256 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148221 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c523fbe-6cf8-456c-acc0-463ff721c1da" containerName="istio-proxy" Oct 31 01:15:00.148256 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148238 4440 cpu_manager.go:414] "containerMap: removing container" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" Oct 31 01:15:00.148256 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148246 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" Oct 31 01:15:00.148256 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148258 4440 cpu_manager.go:414] "containerMap: removing container" podUID="ab8ba016-584a-4646-a3d4-f6db4ccf0700" containerName="payload-pre-processing" Oct 31 01:15:00.148639 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148266 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab8ba016-584a-4646-a3d4-f6db4ccf0700" containerName="payload-pre-processing" Oct 31 01:15:00.148639 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148275 4440 cpu_manager.go:414] "containerMap: removing container" podUID="f889944d-e872-4b84-baa6-829de12f3ba1" containerName="istio-proxy" Oct 31 01:15:00.148639 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148282 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f889944d-e872-4b84-baa6-829de12f3ba1" containerName="istio-proxy" Oct 31 01:15:00.148639 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148295 4440 cpu_manager.go:414] "containerMap: removing container" podUID="be67bf17-2e3c-4624-96cd-aab05f0cdc27" containerName="maas-api" Oct 31 01:15:00.148639 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148301 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be67bf17-2e3c-4624-96cd-aab05f0cdc27" containerName="maas-api" Oct 31 01:15:00.148639 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148309 4440 cpu_manager.go:414] "containerMap: removing container" podUID="b29f36f2-9389-4f3c-91cf-658c373b901c" containerName="istio-proxy" Oct 31 01:15:00.148639 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148316 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b29f36f2-9389-4f3c-91cf-658c373b901c" containerName="istio-proxy" Oct 31 01:15:00.148639 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148327 4440 cpu_manager.go:414] "containerMap: removing container" podUID="f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60" containerName="maas-api" Oct 31 01:15:00.148639 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148336 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60" containerName="maas-api" Oct 31 01:15:00.148639 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148348 4440 cpu_manager.go:414] "containerMap: removing container" podUID="cf86081d-7942-454a-8763-f658bd0ce870" containerName="main" Oct 31 01:15:00.148639 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148354 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf86081d-7942-454a-8763-f658bd0ce870" containerName="main" Oct 31 01:15:00.148639 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148364 4440 cpu_manager.go:414] "containerMap: removing container" podUID="b8df1e49-cfaf-4c8c-9498-3400501ae86b" containerName="payload-processing" Oct 31 01:15:00.148639 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148371 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8df1e49-cfaf-4c8c-9498-3400501ae86b" containerName="payload-processing" Oct 31 01:15:00.148639 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148518 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf86081d-7942-454a-8763-f658bd0ce870" containerName="main" Oct 31 01:15:00.148639 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148533 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8df1e49-cfaf-4c8c-9498-3400501ae86b" containerName="payload-processing" Oct 31 01:15:00.148639 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148547 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc34b9f5-85b1-4cab-a49e-c2106d685cc2" containerName="istio-proxy" Oct 31 01:15:00.148639 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148554 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3" containerName="maas-api" Oct 31 01:15:00.148639 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148566 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f77962d-e14f-4225-b2e9-7a709dc8c4f4" containerName="revoke-keys" Oct 31 01:15:00.148639 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148576 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="583a1183-7c84-40ce-aecb-9663f82a8a47" containerName="istio-proxy" Oct 31 01:15:00.148639 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148586 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="eeb7394c-0c4a-43e6-bbca-f4222bec42ff" containerName="istio-proxy" Oct 31 01:15:00.148639 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148598 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaad3e7d-1293-43d5-9fe6-674e69ea840e" containerName="payload-pre-processing" Oct 31 01:15:00.148639 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148609 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a9a9349-7fdf-4f3a-8f96-62c9134ba1a5" containerName="istio-proxy" Oct 31 01:15:00.148639 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148629 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c523fbe-6cf8-456c-acc0-463ff721c1da" containerName="istio-proxy" Oct 31 01:15:00.148639 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148641 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="f93d4671-a358-4547-ba77-06255ca42c3e" containerName="main" Oct 31 01:15:00.149313 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148651 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab8ba016-584a-4646-a3d4-f6db4ccf0700" containerName="payload-pre-processing" Oct 31 01:15:00.149313 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148658 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="f889944d-e872-4b84-baa6-829de12f3ba1" containerName="istio-proxy" Oct 31 01:15:00.149313 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148666 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="be67bf17-2e3c-4624-96cd-aab05f0cdc27" containerName="maas-api" Oct 31 01:15:00.149313 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148675 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="b29f36f2-9389-4f3c-91cf-658c373b901c" containerName="istio-proxy" Oct 31 01:15:00.149313 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.148685 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2ffbc41-ba1f-47d4-93ab-1b7c1e1e9f60" containerName="maas-api" Oct 31 01:15:00.152627 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.152610 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890155-kdh9z"] Oct 31 01:15:00.152734 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.152719 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-487a6c-29890155-gpd4t" Oct 31 01:15:00.156448 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.156426 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-kdh9z" Oct 31 01:15:00.160920 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.160886 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:15:00.161056 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.160985 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:15:00.162805 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.162772 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890155-kdh9z"] Oct 31 01:15:00.166759 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.165101 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7c55c28-c3ca-4832-9559-b05e99a3995d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890155-kdh9z\" (UID: \"c7c55c28-c3ca-4832-9559-b05e99a3995d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-kdh9z" Oct 31 01:15:00.166759 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.165152 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9nsg\" (UniqueName: \"kubernetes.io/projected/c7c55c28-c3ca-4832-9559-b05e99a3995d-kube-api-access-s9nsg\") pod \"maas-api-key-cleanup-29890155-kdh9z\" (UID: \"c7c55c28-c3ca-4832-9559-b05e99a3995d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-kdh9z" Oct 31 01:15:00.166759 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.165292 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/877c89f5-20ff-4b40-a161-15b6e135c4b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-487a6c-29890155-gpd4t\" (UID: \"877c89f5-20ff-4b40-a161-15b6e135c4b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-487a6c-29890155-gpd4t" Oct 31 01:15:00.166759 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.165329 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjsmv\" (UniqueName: \"kubernetes.io/projected/877c89f5-20ff-4b40-a161-15b6e135c4b9-kube-api-access-tjsmv\") pod \"maas-api-key-cleanup-e2e-worker-w3-487a6c-29890155-gpd4t\" (UID: \"877c89f5-20ff-4b40-a161-15b6e135c4b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-487a6c-29890155-gpd4t" Oct 31 01:15:00.166759 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.166502 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:00.167773 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.167751 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-487a6c-29890155-gpd4t"] Oct 31 01:15:00.183301 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.183002 4440 patch_prober.go:29] interesting pod/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 01:15:00.183301 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.183064 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" podUID="8f8c2432-9405-4f4f-b24f-1b73ea0da86a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 01:15:00.251743 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.251698 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zs6l6\"" Oct 31 01:15:00.266480 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.266444 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/877c89f5-20ff-4b40-a161-15b6e135c4b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-487a6c-29890155-gpd4t\" (UID: \"877c89f5-20ff-4b40-a161-15b6e135c4b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-487a6c-29890155-gpd4t" Oct 31 01:15:00.267440 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.267403 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjsmv\" (UniqueName: \"kubernetes.io/projected/877c89f5-20ff-4b40-a161-15b6e135c4b9-kube-api-access-tjsmv\") pod \"maas-api-key-cleanup-e2e-worker-w3-487a6c-29890155-gpd4t\" (UID: \"877c89f5-20ff-4b40-a161-15b6e135c4b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-487a6c-29890155-gpd4t" Oct 31 01:15:00.267753 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.267736 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7c55c28-c3ca-4832-9559-b05e99a3995d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890155-kdh9z\" (UID: \"c7c55c28-c3ca-4832-9559-b05e99a3995d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-kdh9z" Oct 31 01:15:00.267899 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.267884 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9nsg\" (UniqueName: \"kubernetes.io/projected/c7c55c28-c3ca-4832-9559-b05e99a3995d-kube-api-access-s9nsg\") pod \"maas-api-key-cleanup-29890155-kdh9z\" (UID: \"c7c55c28-c3ca-4832-9559-b05e99a3995d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-kdh9z" Oct 31 01:15:00.268324 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.268035 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/877c89f5-20ff-4b40-a161-15b6e135c4b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-487a6c-29890155-gpd4t\" (UID: \"877c89f5-20ff-4b40-a161-15b6e135c4b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-487a6c-29890155-gpd4t" Oct 31 01:15:00.268513 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.268447 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7c55c28-c3ca-4832-9559-b05e99a3995d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890155-kdh9z\" (UID: \"c7c55c28-c3ca-4832-9559-b05e99a3995d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-kdh9z" Oct 31 01:15:00.277078 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.277025 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjsmv\" (UniqueName: \"kubernetes.io/projected/877c89f5-20ff-4b40-a161-15b6e135c4b9-kube-api-access-tjsmv\") pod \"maas-api-key-cleanup-e2e-worker-w3-487a6c-29890155-gpd4t\" (UID: \"877c89f5-20ff-4b40-a161-15b6e135c4b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-487a6c-29890155-gpd4t" Oct 31 01:15:00.278438 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.278400 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9nsg\" (UniqueName: \"kubernetes.io/projected/c7c55c28-c3ca-4832-9559-b05e99a3995d-kube-api-access-s9nsg\") pod \"maas-api-key-cleanup-29890155-kdh9z\" (UID: \"c7c55c28-c3ca-4832-9559-b05e99a3995d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-kdh9z" Oct 31 01:15:00.379283 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.378374 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" Oct 31 01:15:00.453292 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.452915 4440 patch_prober.go:29] interesting pod/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:15:00.453292 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.452975 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" podUID="81095022-acb2-4b1c-acc6-a9501b0ea478" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:15:00.476040 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.474649 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b4e5ddd-9933-47e1-9265-085858232375-istio-token\" (UniqueName: \"kubernetes.io/projected/3b4e5ddd-9933-47e1-9265-085858232375-istio-token\") pod \"3b4e5ddd-9933-47e1-9265-085858232375\" (UID: \"3b4e5ddd-9933-47e1-9265-085858232375\") " Oct 31 01:15:00.476040 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.474742 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b4e5ddd-9933-47e1-9265-085858232375-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4e5ddd-9933-47e1-9265-085858232375-credential-socket\") pod \"3b4e5ddd-9933-47e1-9265-085858232375\" (UID: \"3b4e5ddd-9933-47e1-9265-085858232375\") " Oct 31 01:15:00.476040 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.474813 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b4e5ddd-9933-47e1-9265-085858232375-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b4e5ddd-9933-47e1-9265-085858232375-istio-envoy\") pod \"3b4e5ddd-9933-47e1-9265-085858232375\" (UID: \"3b4e5ddd-9933-47e1-9265-085858232375\") " Oct 31 01:15:00.476040 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.474913 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b4e5ddd-9933-47e1-9265-085858232375-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b4e5ddd-9933-47e1-9265-085858232375-istiod-ca-cert\") pod \"3b4e5ddd-9933-47e1-9265-085858232375\" (UID: \"3b4e5ddd-9933-47e1-9265-085858232375\") " Oct 31 01:15:00.476040 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.474953 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b4e5ddd-9933-47e1-9265-085858232375-kube-api-access-qnj6k\" (UniqueName: \"kubernetes.io/projected/3b4e5ddd-9933-47e1-9265-085858232375-kube-api-access-qnj6k\") pod \"3b4e5ddd-9933-47e1-9265-085858232375\" (UID: \"3b4e5ddd-9933-47e1-9265-085858232375\") " Oct 31 01:15:00.476040 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.474988 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b4e5ddd-9933-47e1-9265-085858232375-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b4e5ddd-9933-47e1-9265-085858232375-workload-certs\") pod \"3b4e5ddd-9933-47e1-9265-085858232375\" (UID: \"3b4e5ddd-9933-47e1-9265-085858232375\") " Oct 31 01:15:00.476040 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.475027 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b4e5ddd-9933-47e1-9265-085858232375-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4e5ddd-9933-47e1-9265-085858232375-workload-socket\") pod \"3b4e5ddd-9933-47e1-9265-085858232375\" (UID: \"3b4e5ddd-9933-47e1-9265-085858232375\") " Oct 31 01:15:00.476040 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.475085 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b4e5ddd-9933-47e1-9265-085858232375-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b4e5ddd-9933-47e1-9265-085858232375-istio-data\") pod \"3b4e5ddd-9933-47e1-9265-085858232375\" (UID: \"3b4e5ddd-9933-47e1-9265-085858232375\") " Oct 31 01:15:00.476040 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.475183 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b4e5ddd-9933-47e1-9265-085858232375-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b4e5ddd-9933-47e1-9265-085858232375-istio-podinfo\") pod \"3b4e5ddd-9933-47e1-9265-085858232375\" (UID: \"3b4e5ddd-9933-47e1-9265-085858232375\") " Oct 31 01:15:00.478213 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.478181 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b4e5ddd-9933-47e1-9265-085858232375-istiod-ca-cert" pod "3b4e5ddd-9933-47e1-9265-085858232375" (UID: "3b4e5ddd-9933-47e1-9265-085858232375"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:00.479642 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.479552 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b4e5ddd-9933-47e1-9265-085858232375-istio-data" pod "3b4e5ddd-9933-47e1-9265-085858232375" (UID: "3b4e5ddd-9933-47e1-9265-085858232375"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:00.479642 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.479619 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b4e5ddd-9933-47e1-9265-085858232375-credential-socket" pod "3b4e5ddd-9933-47e1-9265-085858232375" (UID: "3b4e5ddd-9933-47e1-9265-085858232375"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:00.480005 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.479981 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b4e5ddd-9933-47e1-9265-085858232375-workload-socket" pod "3b4e5ddd-9933-47e1-9265-085858232375" (UID: "3b4e5ddd-9933-47e1-9265-085858232375"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:00.480242 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.480227 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b4e5ddd-9933-47e1-9265-085858232375-workload-certs" pod "3b4e5ddd-9933-47e1-9265-085858232375" (UID: "3b4e5ddd-9933-47e1-9265-085858232375"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:00.492955 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.492916 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b4e5ddd-9933-47e1-9265-085858232375-istio-token" pod "3b4e5ddd-9933-47e1-9265-085858232375" (UID: "3b4e5ddd-9933-47e1-9265-085858232375"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:00.499082 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.499047 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b4e5ddd-9933-47e1-9265-085858232375-istio-envoy" pod "3b4e5ddd-9933-47e1-9265-085858232375" (UID: "3b4e5ddd-9933-47e1-9265-085858232375"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:00.499705 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.499677 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b4e5ddd-9933-47e1-9265-085858232375-kube-api-access-qnj6k" pod "3b4e5ddd-9933-47e1-9265-085858232375" (UID: "3b4e5ddd-9933-47e1-9265-085858232375"). InnerVolumeSpecName "kube-api-access-qnj6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:00.520859 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.520820 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b4e5ddd-9933-47e1-9265-085858232375-istio-podinfo" pod "3b4e5ddd-9933-47e1-9265-085858232375" (UID: "3b4e5ddd-9933-47e1-9265-085858232375"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:15:00.553967 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.553938 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-487a6c-29890155-gpd4t" Oct 31 01:15:00.556644 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.556406 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" Oct 31 01:15:00.560649 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.560622 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" Oct 31 01:15:00.567931 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.564629 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-kdh9z" Oct 31 01:15:00.576822 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.576791 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-workload-certs\") pod \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\" (UID: \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\") " Oct 31 01:15:00.576937 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.576859 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-credential-socket\") pod \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\" (UID: \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\") " Oct 31 01:15:00.576937 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.576890 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-kube-api-access-ddwsg\" (UniqueName: \"kubernetes.io/projected/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-kube-api-access-ddwsg\") pod \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\" (UID: \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\") " Oct 31 01:15:00.577014 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.576936 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b192b955-7018-4d48-b3c6-c58d86eeb5b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b192b955-7018-4d48-b3c6-c58d86eeb5b2-istio-data\") pod \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\" (UID: \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\") " Oct 31 01:15:00.577014 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.576998 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b192b955-7018-4d48-b3c6-c58d86eeb5b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b192b955-7018-4d48-b3c6-c58d86eeb5b2-workload-socket\") pod \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\" (UID: \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\") " Oct 31 01:15:00.577086 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.577026 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b192b955-7018-4d48-b3c6-c58d86eeb5b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b192b955-7018-4d48-b3c6-c58d86eeb5b2-workload-certs\") pod \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\" (UID: \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\") " Oct 31 01:15:00.577137 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.577123 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-istio-token\" (UniqueName: \"kubernetes.io/projected/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-istio-token\") pod \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\" (UID: \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\") " Oct 31 01:15:00.577172 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.577154 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-istio-envoy\") pod \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\" (UID: \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\") " Oct 31 01:15:00.577213 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.577187 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b192b955-7018-4d48-b3c6-c58d86eeb5b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b192b955-7018-4d48-b3c6-c58d86eeb5b2-istio-envoy\") pod \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\" (UID: \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\") " Oct 31 01:15:00.577259 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.577244 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-istio-data\") pod \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\" (UID: \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\") " Oct 31 01:15:00.577298 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.577279 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b192b955-7018-4d48-b3c6-c58d86eeb5b2-istio-token\" (UniqueName: \"kubernetes.io/projected/b192b955-7018-4d48-b3c6-c58d86eeb5b2-istio-token\") pod \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\" (UID: \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\") " Oct 31 01:15:00.577334 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.577325 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-istio-podinfo\") pod \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\" (UID: \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\") " Oct 31 01:15:00.577381 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.577367 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-workload-socket\") pod \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\" (UID: \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\") " Oct 31 01:15:00.577493 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.577449 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b192b955-7018-4d48-b3c6-c58d86eeb5b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b192b955-7018-4d48-b3c6-c58d86eeb5b2-istio-podinfo\") pod \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\" (UID: \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\") " Oct 31 01:15:00.577554 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.577519 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-istiod-ca-cert\") pod \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\" (UID: \"8f8c2432-9405-4f4f-b24f-1b73ea0da86a\") " Oct 31 01:15:00.577594 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.577558 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b192b955-7018-4d48-b3c6-c58d86eeb5b2-kube-api-access-gpb6q\" (UniqueName: \"kubernetes.io/projected/b192b955-7018-4d48-b3c6-c58d86eeb5b2-kube-api-access-gpb6q\") pod \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\" (UID: \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\") " Oct 31 01:15:00.577639 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.577611 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b192b955-7018-4d48-b3c6-c58d86eeb5b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b192b955-7018-4d48-b3c6-c58d86eeb5b2-credential-socket\") pod \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\" (UID: \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\") " Oct 31 01:15:00.577685 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.577674 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b192b955-7018-4d48-b3c6-c58d86eeb5b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b192b955-7018-4d48-b3c6-c58d86eeb5b2-istiod-ca-cert\") pod \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\" (UID: \"b192b955-7018-4d48-b3c6-c58d86eeb5b2\") " Oct 31 01:15:00.578471 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.578010 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-workload-socket" pod "8f8c2432-9405-4f4f-b24f-1b73ea0da86a" (UID: "8f8c2432-9405-4f4f-b24f-1b73ea0da86a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:00.578471 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.578174 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-istio-data" pod "8f8c2432-9405-4f4f-b24f-1b73ea0da86a" (UID: "8f8c2432-9405-4f4f-b24f-1b73ea0da86a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:00.578471 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.578457 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-istiod-ca-cert" pod "8f8c2432-9405-4f4f-b24f-1b73ea0da86a" (UID: "8f8c2432-9405-4f4f-b24f-1b73ea0da86a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:00.578800 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.578778 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b192b955-7018-4d48-b3c6-c58d86eeb5b2-istiod-ca-cert" pod "b192b955-7018-4d48-b3c6-c58d86eeb5b2" (UID: "b192b955-7018-4d48-b3c6-c58d86eeb5b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:00.578926 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.578904 4440 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b4e5ddd-9933-47e1-9265-085858232375-istio-envoy\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:00.578961 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.578935 4440 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b192b955-7018-4d48-b3c6-c58d86eeb5b2-istiod-ca-cert\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:00.578961 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.578951 4440 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b4e5ddd-9933-47e1-9265-085858232375-istiod-ca-cert\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:00.579067 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.578966 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnj6k\" (UniqueName: \"kubernetes.io/projected/3b4e5ddd-9933-47e1-9265-085858232375-kube-api-access-qnj6k\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:00.579067 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.578983 4440 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b4e5ddd-9933-47e1-9265-085858232375-workload-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:00.579067 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.578998 4440 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4e5ddd-9933-47e1-9265-085858232375-workload-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:00.579067 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.579017 4440 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b4e5ddd-9933-47e1-9265-085858232375-istio-data\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:00.579067 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.579033 4440 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b4e5ddd-9933-47e1-9265-085858232375-istio-podinfo\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:00.579067 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.579047 4440 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-istio-data\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:00.579067 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.579060 4440 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b4e5ddd-9933-47e1-9265-085858232375-istio-token\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:00.579397 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.579075 4440 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-workload-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:00.579397 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.579090 4440 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4e5ddd-9933-47e1-9265-085858232375-credential-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:00.579397 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.579104 4440 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-istiod-ca-cert\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:00.579585 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.579566 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b192b955-7018-4d48-b3c6-c58d86eeb5b2-istio-data" pod "b192b955-7018-4d48-b3c6-c58d86eeb5b2" (UID: "b192b955-7018-4d48-b3c6-c58d86eeb5b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:00.579794 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.579771 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-credential-socket" pod "8f8c2432-9405-4f4f-b24f-1b73ea0da86a" (UID: "8f8c2432-9405-4f4f-b24f-1b73ea0da86a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:00.582241 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.580151 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b192b955-7018-4d48-b3c6-c58d86eeb5b2-workload-certs" pod "b192b955-7018-4d48-b3c6-c58d86eeb5b2" (UID: "b192b955-7018-4d48-b3c6-c58d86eeb5b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:00.582241 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.580342 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b192b955-7018-4d48-b3c6-c58d86eeb5b2-workload-socket" pod "b192b955-7018-4d48-b3c6-c58d86eeb5b2" (UID: "b192b955-7018-4d48-b3c6-c58d86eeb5b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:00.582241 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.580495 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b192b955-7018-4d48-b3c6-c58d86eeb5b2-credential-socket" pod "b192b955-7018-4d48-b3c6-c58d86eeb5b2" (UID: "b192b955-7018-4d48-b3c6-c58d86eeb5b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:00.582241 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.580528 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-workload-certs" pod "8f8c2432-9405-4f4f-b24f-1b73ea0da86a" (UID: "8f8c2432-9405-4f4f-b24f-1b73ea0da86a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:00.583268 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.583219 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b192b955-7018-4d48-b3c6-c58d86eeb5b2-kube-api-access-gpb6q" pod "b192b955-7018-4d48-b3c6-c58d86eeb5b2" (UID: "b192b955-7018-4d48-b3c6-c58d86eeb5b2"). InnerVolumeSpecName "kube-api-access-gpb6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:00.586626 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.586587 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-istio-podinfo" pod "8f8c2432-9405-4f4f-b24f-1b73ea0da86a" (UID: "8f8c2432-9405-4f4f-b24f-1b73ea0da86a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:15:00.586836 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.586807 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-istio-envoy" pod "8f8c2432-9405-4f4f-b24f-1b73ea0da86a" (UID: "8f8c2432-9405-4f4f-b24f-1b73ea0da86a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:00.587261 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.587228 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b192b955-7018-4d48-b3c6-c58d86eeb5b2-istio-podinfo" pod "b192b955-7018-4d48-b3c6-c58d86eeb5b2" (UID: "b192b955-7018-4d48-b3c6-c58d86eeb5b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:15:00.587261 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.587237 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b192b955-7018-4d48-b3c6-c58d86eeb5b2-istio-envoy" pod "b192b955-7018-4d48-b3c6-c58d86eeb5b2" (UID: "b192b955-7018-4d48-b3c6-c58d86eeb5b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:00.588989 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.588955 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-istio-token" pod "8f8c2432-9405-4f4f-b24f-1b73ea0da86a" (UID: "8f8c2432-9405-4f4f-b24f-1b73ea0da86a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:00.590559 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.589612 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b192b955-7018-4d48-b3c6-c58d86eeb5b2-istio-token" pod "b192b955-7018-4d48-b3c6-c58d86eeb5b2" (UID: "b192b955-7018-4d48-b3c6-c58d86eeb5b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:00.590559 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.587249 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-kube-api-access-ddwsg" pod "8f8c2432-9405-4f4f-b24f-1b73ea0da86a" (UID: "8f8c2432-9405-4f4f-b24f-1b73ea0da86a"). InnerVolumeSpecName "kube-api-access-ddwsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:00.622299 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.621991 4440 patch_prober.go:29] interesting pod/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:15:00.622299 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.622051 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" podUID="7ee3eb69-94de-447c-b9ec-194a0458f2e7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:15:00.684584 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.684472 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:00.685600 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.685349 4440 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-istio-token\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:00.685600 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.685386 4440 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-istio-envoy\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:00.685600 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.685402 4440 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b192b955-7018-4d48-b3c6-c58d86eeb5b2-istio-envoy\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:00.685600 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.685432 4440 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b192b955-7018-4d48-b3c6-c58d86eeb5b2-istio-token\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:00.685600 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.685448 4440 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-istio-podinfo\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:00.685600 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.685462 4440 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b192b955-7018-4d48-b3c6-c58d86eeb5b2-istio-podinfo\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:00.685600 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.685477 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpb6q\" (UniqueName: \"kubernetes.io/projected/b192b955-7018-4d48-b3c6-c58d86eeb5b2-kube-api-access-gpb6q\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:00.685600 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.685492 4440 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b192b955-7018-4d48-b3c6-c58d86eeb5b2-credential-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:00.685600 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.685506 4440 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-workload-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:00.685600 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.685519 4440 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-credential-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:00.685600 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.685534 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddwsg\" (UniqueName: \"kubernetes.io/projected/8f8c2432-9405-4f4f-b24f-1b73ea0da86a-kube-api-access-ddwsg\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:00.685600 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.685547 4440 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b192b955-7018-4d48-b3c6-c58d86eeb5b2-istio-data\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:00.685600 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.685561 4440 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b192b955-7018-4d48-b3c6-c58d86eeb5b2-workload-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:00.685600 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.685574 4440 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b192b955-7018-4d48-b3c6-c58d86eeb5b2-workload-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:00.879044 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.879009 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-487a6c-29890155-gpd4t"] Oct 31 01:15:00.880719 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:15:00.880688 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod877c89f5_20ff_4b40_a161_15b6e135c4b9.slice/crio-7c0ee9ae135a3430cd3268a08780c0c084703fc68762ed925210d16c6614cbaa WatchSource:0}: Error finding container 7c0ee9ae135a3430cd3268a08780c0c084703fc68762ed925210d16c6614cbaa: Status 404 returned error can't find the container with id 7c0ee9ae135a3430cd3268a08780c0c084703fc68762ed925210d16c6614cbaa Oct 31 01:15:00.890065 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.890040 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890155-kdh9z"] Oct 31 01:15:00.893027 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:15:00.893004 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7c55c28_c3ca_4832_9559_b05e99a3995d.slice/crio-4a0a83afd9685fed4b8892cc23314549b85cdb0274712b361bd95994e7ad61d0 WatchSource:0}: Error finding container 4a0a83afd9685fed4b8892cc23314549b85cdb0274712b361bd95994e7ad61d0: Status 404 returned error can't find the container with id 4a0a83afd9685fed4b8892cc23314549b85cdb0274712b361bd95994e7ad61d0 Oct 31 01:15:00.929656 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.929626 4440 generic.go:356] "Generic (PLEG): container finished" podID="8f8c2432-9405-4f4f-b24f-1b73ea0da86a" containerID="3268df394c99ef1132a6a6b2280a81753857a6722c453d7681cb7f6e93d56da6" exitCode=0 Oct 31 01:15:00.929805 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.929676 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" Oct 31 01:15:00.929805 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.929702 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" event={"ID":"8f8c2432-9405-4f4f-b24f-1b73ea0da86a","Type":"ContainerDied","Data":"3268df394c99ef1132a6a6b2280a81753857a6722c453d7681cb7f6e93d56da6"} Oct 31 01:15:00.929805 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.929750 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh" event={"ID":"8f8c2432-9405-4f4f-b24f-1b73ea0da86a","Type":"ContainerDied","Data":"0efd785b943e9ba0cb070853bb337f6f6df533d39460ee9055512b1e7832b307"} Oct 31 01:15:00.929805 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.929778 4440 scope.go:122] "RemoveContainer" containerID="3268df394c99ef1132a6a6b2280a81753857a6722c453d7681cb7f6e93d56da6" Oct 31 01:15:00.931262 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.931239 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-487a6c-29890155-gpd4t" event={"ID":"877c89f5-20ff-4b40-a161-15b6e135c4b9","Type":"ContainerStarted","Data":"7c0ee9ae135a3430cd3268a08780c0c084703fc68762ed925210d16c6614cbaa"} Oct 31 01:15:00.932877 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.932853 4440 generic.go:356] "Generic (PLEG): container finished" podID="3b4e5ddd-9933-47e1-9265-085858232375" containerID="9159b732b08f6092f3f7cf448af44053edff619b23624a91e1b5ae0f215011e0" exitCode=0 Oct 31 01:15:00.932995 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.932914 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" Oct 31 01:15:00.932995 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.932926 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" event={"ID":"3b4e5ddd-9933-47e1-9265-085858232375","Type":"ContainerDied","Data":"9159b732b08f6092f3f7cf448af44053edff619b23624a91e1b5ae0f215011e0"} Oct 31 01:15:00.933133 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.932999 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4" event={"ID":"3b4e5ddd-9933-47e1-9265-085858232375","Type":"ContainerDied","Data":"f5013138e7196fda65733f7931aade76b2865fa9ad1eab8a0e94266177487e3d"} Oct 31 01:15:00.934334 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.934315 4440 generic.go:356] "Generic (PLEG): container finished" podID="b192b955-7018-4d48-b3c6-c58d86eeb5b2" containerID="8a419a62c34d493bacbfce57925ee0e68c8d090600939ab0e06825ba43becc17" exitCode=0 Oct 31 01:15:00.934456 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.934400 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" event={"ID":"b192b955-7018-4d48-b3c6-c58d86eeb5b2","Type":"ContainerDied","Data":"8a419a62c34d493bacbfce57925ee0e68c8d090600939ab0e06825ba43becc17"} Oct 31 01:15:00.934456 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.934439 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" Oct 31 01:15:00.934570 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.934462 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph" event={"ID":"b192b955-7018-4d48-b3c6-c58d86eeb5b2","Type":"ContainerDied","Data":"bcab6fb945016f475c917bb990350de5fe1eb8e07f36ab701d4e351c79f3de18"} Oct 31 01:15:00.938946 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.938912 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-kdh9z" event={"ID":"c7c55c28-c3ca-4832-9559-b05e99a3995d","Type":"ContainerStarted","Data":"4a0a83afd9685fed4b8892cc23314549b85cdb0274712b361bd95994e7ad61d0"} Oct 31 01:15:00.945779 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.945758 4440 scope.go:122] "RemoveContainer" containerID="3268df394c99ef1132a6a6b2280a81753857a6722c453d7681cb7f6e93d56da6" Oct 31 01:15:00.946111 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:15:00.946093 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3268df394c99ef1132a6a6b2280a81753857a6722c453d7681cb7f6e93d56da6\": container with ID starting with 3268df394c99ef1132a6a6b2280a81753857a6722c453d7681cb7f6e93d56da6 not found: ID does not exist" containerID="3268df394c99ef1132a6a6b2280a81753857a6722c453d7681cb7f6e93d56da6" Oct 31 01:15:00.946166 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.946119 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3268df394c99ef1132a6a6b2280a81753857a6722c453d7681cb7f6e93d56da6"} err="failed to get container status \"3268df394c99ef1132a6a6b2280a81753857a6722c453d7681cb7f6e93d56da6\": rpc error: code = NotFound desc = could not find container \"3268df394c99ef1132a6a6b2280a81753857a6722c453d7681cb7f6e93d56da6\": container with ID starting with 3268df394c99ef1132a6a6b2280a81753857a6722c453d7681cb7f6e93d56da6 not found: ID does not exist" Oct 31 01:15:00.946166 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.946136 4440 scope.go:122] "RemoveContainer" containerID="9159b732b08f6092f3f7cf448af44053edff619b23624a91e1b5ae0f215011e0" Oct 31 01:15:00.962105 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.961969 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4"] Oct 31 01:15:00.962332 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.962307 4440 scope.go:122] "RemoveContainer" containerID="9159b732b08f6092f3f7cf448af44053edff619b23624a91e1b5ae0f215011e0" Oct 31 01:15:00.962733 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:15:00.962711 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9159b732b08f6092f3f7cf448af44053edff619b23624a91e1b5ae0f215011e0\": container with ID starting with 9159b732b08f6092f3f7cf448af44053edff619b23624a91e1b5ae0f215011e0 not found: ID does not exist" containerID="9159b732b08f6092f3f7cf448af44053edff619b23624a91e1b5ae0f215011e0" Oct 31 01:15:00.962836 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.962741 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9159b732b08f6092f3f7cf448af44053edff619b23624a91e1b5ae0f215011e0"} err="failed to get container status \"9159b732b08f6092f3f7cf448af44053edff619b23624a91e1b5ae0f215011e0\": rpc error: code = NotFound desc = could not find container \"9159b732b08f6092f3f7cf448af44053edff619b23624a91e1b5ae0f215011e0\": container with ID starting with 9159b732b08f6092f3f7cf448af44053edff619b23624a91e1b5ae0f215011e0 not found: ID does not exist" Oct 31 01:15:00.962836 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.962759 4440 scope.go:122] "RemoveContainer" containerID="8a419a62c34d493bacbfce57925ee0e68c8d090600939ab0e06825ba43becc17" Oct 31 01:15:00.966189 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.966163 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-hwbm4"] Oct 31 01:15:00.974003 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.973984 4440 scope.go:122] "RemoveContainer" containerID="8a419a62c34d493bacbfce57925ee0e68c8d090600939ab0e06825ba43becc17" Oct 31 01:15:00.974282 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:15:00.974260 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a419a62c34d493bacbfce57925ee0e68c8d090600939ab0e06825ba43becc17\": container with ID starting with 8a419a62c34d493bacbfce57925ee0e68c8d090600939ab0e06825ba43becc17 not found: ID does not exist" containerID="8a419a62c34d493bacbfce57925ee0e68c8d090600939ab0e06825ba43becc17" Oct 31 01:15:00.974345 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.974294 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a419a62c34d493bacbfce57925ee0e68c8d090600939ab0e06825ba43becc17"} err="failed to get container status \"8a419a62c34d493bacbfce57925ee0e68c8d090600939ab0e06825ba43becc17\": rpc error: code = NotFound desc = could not find container \"8a419a62c34d493bacbfce57925ee0e68c8d090600939ab0e06825ba43becc17\": container with ID starting with 8a419a62c34d493bacbfce57925ee0e68c8d090600939ab0e06825ba43becc17 not found: ID does not exist" Oct 31 01:15:00.976058 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.976038 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh"] Oct 31 01:15:00.987451 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:00.987401 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-wxvvh"] Oct 31 01:15:01.000973 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.000938 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph"] Oct 31 01:15:01.006862 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.006829 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7f6d1e-openshift-default-67d6894945-4xpph"] Oct 31 01:15:01.021243 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:15:01.021199 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:15:01.423552 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.423516 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" Oct 31 01:15:01.460231 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.460195 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:01.503487 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.496868 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f420497-f98d-4a0c-8552-fca5c3f002f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f420497-f98d-4a0c-8552-fca5c3f002f2-workload-certs\") pod \"5f420497-f98d-4a0c-8552-fca5c3f002f2\" (UID: \"5f420497-f98d-4a0c-8552-fca5c3f002f2\") " Oct 31 01:15:01.503487 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.496993 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f420497-f98d-4a0c-8552-fca5c3f002f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f420497-f98d-4a0c-8552-fca5c3f002f2-credential-socket\") pod \"5f420497-f98d-4a0c-8552-fca5c3f002f2\" (UID: \"5f420497-f98d-4a0c-8552-fca5c3f002f2\") " Oct 31 01:15:01.503487 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.497046 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f420497-f98d-4a0c-8552-fca5c3f002f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f420497-f98d-4a0c-8552-fca5c3f002f2-istiod-ca-cert\") pod \"5f420497-f98d-4a0c-8552-fca5c3f002f2\" (UID: \"5f420497-f98d-4a0c-8552-fca5c3f002f2\") " Oct 31 01:15:01.503487 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.497077 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f420497-f98d-4a0c-8552-fca5c3f002f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f420497-f98d-4a0c-8552-fca5c3f002f2-workload-socket\") pod \"5f420497-f98d-4a0c-8552-fca5c3f002f2\" (UID: \"5f420497-f98d-4a0c-8552-fca5c3f002f2\") " Oct 31 01:15:01.503487 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.497153 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f420497-f98d-4a0c-8552-fca5c3f002f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f420497-f98d-4a0c-8552-fca5c3f002f2-istio-data\") pod \"5f420497-f98d-4a0c-8552-fca5c3f002f2\" (UID: \"5f420497-f98d-4a0c-8552-fca5c3f002f2\") " Oct 31 01:15:01.503487 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.497196 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f420497-f98d-4a0c-8552-fca5c3f002f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f420497-f98d-4a0c-8552-fca5c3f002f2-istio-envoy\") pod \"5f420497-f98d-4a0c-8552-fca5c3f002f2\" (UID: \"5f420497-f98d-4a0c-8552-fca5c3f002f2\") " Oct 31 01:15:01.503487 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.497240 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f420497-f98d-4a0c-8552-fca5c3f002f2-istio-token\" (UniqueName: \"kubernetes.io/projected/5f420497-f98d-4a0c-8552-fca5c3f002f2-istio-token\") pod \"5f420497-f98d-4a0c-8552-fca5c3f002f2\" (UID: \"5f420497-f98d-4a0c-8552-fca5c3f002f2\") " Oct 31 01:15:01.503487 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.497275 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f420497-f98d-4a0c-8552-fca5c3f002f2-kube-api-access-tsh4t\" (UniqueName: \"kubernetes.io/projected/5f420497-f98d-4a0c-8552-fca5c3f002f2-kube-api-access-tsh4t\") pod \"5f420497-f98d-4a0c-8552-fca5c3f002f2\" (UID: \"5f420497-f98d-4a0c-8552-fca5c3f002f2\") " Oct 31 01:15:01.503487 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.497314 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f420497-f98d-4a0c-8552-fca5c3f002f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f420497-f98d-4a0c-8552-fca5c3f002f2-istio-podinfo\") pod \"5f420497-f98d-4a0c-8552-fca5c3f002f2\" (UID: \"5f420497-f98d-4a0c-8552-fca5c3f002f2\") " Oct 31 01:15:01.503487 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.499466 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f420497-f98d-4a0c-8552-fca5c3f002f2-workload-certs" pod "5f420497-f98d-4a0c-8552-fca5c3f002f2" (UID: "5f420497-f98d-4a0c-8552-fca5c3f002f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:01.503487 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.499889 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f420497-f98d-4a0c-8552-fca5c3f002f2-istiod-ca-cert" pod "5f420497-f98d-4a0c-8552-fca5c3f002f2" (UID: "5f420497-f98d-4a0c-8552-fca5c3f002f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:01.503487 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.499984 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f420497-f98d-4a0c-8552-fca5c3f002f2-credential-socket" pod "5f420497-f98d-4a0c-8552-fca5c3f002f2" (UID: "5f420497-f98d-4a0c-8552-fca5c3f002f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:01.503487 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.500215 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f420497-f98d-4a0c-8552-fca5c3f002f2-workload-socket" pod "5f420497-f98d-4a0c-8552-fca5c3f002f2" (UID: "5f420497-f98d-4a0c-8552-fca5c3f002f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:01.503487 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.500387 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f420497-f98d-4a0c-8552-fca5c3f002f2-istio-data" pod "5f420497-f98d-4a0c-8552-fca5c3f002f2" (UID: "5f420497-f98d-4a0c-8552-fca5c3f002f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:01.541001 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.540623 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" Oct 31 01:15:01.601861 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.598568 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f420497-f98d-4a0c-8552-fca5c3f002f2-kube-api-access-tsh4t" pod "5f420497-f98d-4a0c-8552-fca5c3f002f2" (UID: "5f420497-f98d-4a0c-8552-fca5c3f002f2"). InnerVolumeSpecName "kube-api-access-tsh4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:01.601861 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.598868 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81095022-acb2-4b1c-acc6-a9501b0ea478-istio-token\" (UniqueName: \"kubernetes.io/projected/81095022-acb2-4b1c-acc6-a9501b0ea478-istio-token\") pod \"81095022-acb2-4b1c-acc6-a9501b0ea478\" (UID: \"81095022-acb2-4b1c-acc6-a9501b0ea478\") " Oct 31 01:15:01.601861 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.598993 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81095022-acb2-4b1c-acc6-a9501b0ea478-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81095022-acb2-4b1c-acc6-a9501b0ea478-workload-socket\") pod \"81095022-acb2-4b1c-acc6-a9501b0ea478\" (UID: \"81095022-acb2-4b1c-acc6-a9501b0ea478\") " Oct 31 01:15:01.601861 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.599078 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81095022-acb2-4b1c-acc6-a9501b0ea478-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81095022-acb2-4b1c-acc6-a9501b0ea478-workload-certs\") pod \"81095022-acb2-4b1c-acc6-a9501b0ea478\" (UID: \"81095022-acb2-4b1c-acc6-a9501b0ea478\") " Oct 31 01:15:01.601861 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.599114 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81095022-acb2-4b1c-acc6-a9501b0ea478-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81095022-acb2-4b1c-acc6-a9501b0ea478-istio-podinfo\") pod \"81095022-acb2-4b1c-acc6-a9501b0ea478\" (UID: \"81095022-acb2-4b1c-acc6-a9501b0ea478\") " Oct 31 01:15:01.601861 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.599144 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f420497-f98d-4a0c-8552-fca5c3f002f2-kube-api-access-tsh4t\" (UniqueName: \"kubernetes.io/projected/5f420497-f98d-4a0c-8552-fca5c3f002f2-kube-api-access-tsh4t\") pod \"5f420497-f98d-4a0c-8552-fca5c3f002f2\" (UID: \"5f420497-f98d-4a0c-8552-fca5c3f002f2\") " Oct 31 01:15:01.601861 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.599181 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81095022-acb2-4b1c-acc6-a9501b0ea478-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81095022-acb2-4b1c-acc6-a9501b0ea478-credential-socket\") pod \"81095022-acb2-4b1c-acc6-a9501b0ea478\" (UID: \"81095022-acb2-4b1c-acc6-a9501b0ea478\") " Oct 31 01:15:01.601861 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.599216 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81095022-acb2-4b1c-acc6-a9501b0ea478-kube-api-access-66kb5\" (UniqueName: \"kubernetes.io/projected/81095022-acb2-4b1c-acc6-a9501b0ea478-kube-api-access-66kb5\") pod \"81095022-acb2-4b1c-acc6-a9501b0ea478\" (UID: \"81095022-acb2-4b1c-acc6-a9501b0ea478\") " Oct 31 01:15:01.601861 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.599275 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81095022-acb2-4b1c-acc6-a9501b0ea478-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81095022-acb2-4b1c-acc6-a9501b0ea478-istio-envoy\") pod \"81095022-acb2-4b1c-acc6-a9501b0ea478\" (UID: \"81095022-acb2-4b1c-acc6-a9501b0ea478\") " Oct 31 01:15:01.601861 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.599328 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81095022-acb2-4b1c-acc6-a9501b0ea478-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81095022-acb2-4b1c-acc6-a9501b0ea478-istiod-ca-cert\") pod \"81095022-acb2-4b1c-acc6-a9501b0ea478\" (UID: \"81095022-acb2-4b1c-acc6-a9501b0ea478\") " Oct 31 01:15:01.601861 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.599394 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81095022-acb2-4b1c-acc6-a9501b0ea478-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81095022-acb2-4b1c-acc6-a9501b0ea478-istio-data\") pod \"81095022-acb2-4b1c-acc6-a9501b0ea478\" (UID: \"81095022-acb2-4b1c-acc6-a9501b0ea478\") " Oct 31 01:15:01.601861 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.599842 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81095022-acb2-4b1c-acc6-a9501b0ea478-workload-socket" pod "81095022-acb2-4b1c-acc6-a9501b0ea478" (UID: "81095022-acb2-4b1c-acc6-a9501b0ea478"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:01.601861 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.600057 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81095022-acb2-4b1c-acc6-a9501b0ea478-istio-data" pod "81095022-acb2-4b1c-acc6-a9501b0ea478" (UID: "81095022-acb2-4b1c-acc6-a9501b0ea478"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:01.601861 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.600117 4440 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f420497-f98d-4a0c-8552-fca5c3f002f2-workload-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:01.601861 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.600142 4440 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f420497-f98d-4a0c-8552-fca5c3f002f2-credential-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:01.601861 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.600158 4440 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f420497-f98d-4a0c-8552-fca5c3f002f2-istiod-ca-cert\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:01.601861 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.600173 4440 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f420497-f98d-4a0c-8552-fca5c3f002f2-workload-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:01.601861 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.600188 4440 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81095022-acb2-4b1c-acc6-a9501b0ea478-workload-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:01.601861 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.600200 4440 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f420497-f98d-4a0c-8552-fca5c3f002f2-istio-data\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:01.601861 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:15:01.600124 4440 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/5f420497-f98d-4a0c-8552-fca5c3f002f2/volumes/kubernetes.io~projected/kube-api-access-tsh4t Oct 31 01:15:01.601861 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.600247 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f420497-f98d-4a0c-8552-fca5c3f002f2-kube-api-access-tsh4t" pod "5f420497-f98d-4a0c-8552-fca5c3f002f2" (UID: "5f420497-f98d-4a0c-8552-fca5c3f002f2"). InnerVolumeSpecName "kube-api-access-tsh4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:01.601861 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.600289 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81095022-acb2-4b1c-acc6-a9501b0ea478-credential-socket" pod "81095022-acb2-4b1c-acc6-a9501b0ea478" (UID: "81095022-acb2-4b1c-acc6-a9501b0ea478"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:01.601861 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.600469 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81095022-acb2-4b1c-acc6-a9501b0ea478-workload-certs" pod "81095022-acb2-4b1c-acc6-a9501b0ea478" (UID: "81095022-acb2-4b1c-acc6-a9501b0ea478"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:01.601861 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.600951 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f420497-f98d-4a0c-8552-fca5c3f002f2-istio-podinfo" pod "5f420497-f98d-4a0c-8552-fca5c3f002f2" (UID: "5f420497-f98d-4a0c-8552-fca5c3f002f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:15:01.601861 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.601060 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f420497-f98d-4a0c-8552-fca5c3f002f2-istio-envoy" pod "5f420497-f98d-4a0c-8552-fca5c3f002f2" (UID: "5f420497-f98d-4a0c-8552-fca5c3f002f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:01.601861 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.601886 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81095022-acb2-4b1c-acc6-a9501b0ea478-istiod-ca-cert" pod "81095022-acb2-4b1c-acc6-a9501b0ea478" (UID: "81095022-acb2-4b1c-acc6-a9501b0ea478"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:01.608807 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.608746 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81095022-acb2-4b1c-acc6-a9501b0ea478-kube-api-access-66kb5" pod "81095022-acb2-4b1c-acc6-a9501b0ea478" (UID: "81095022-acb2-4b1c-acc6-a9501b0ea478"). InnerVolumeSpecName "kube-api-access-66kb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:01.609172 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.609122 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81095022-acb2-4b1c-acc6-a9501b0ea478-istio-podinfo" pod "81095022-acb2-4b1c-acc6-a9501b0ea478" (UID: "81095022-acb2-4b1c-acc6-a9501b0ea478"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:15:01.609343 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.609175 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81095022-acb2-4b1c-acc6-a9501b0ea478-istio-token" pod "81095022-acb2-4b1c-acc6-a9501b0ea478" (UID: "81095022-acb2-4b1c-acc6-a9501b0ea478"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:01.610955 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.610922 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81095022-acb2-4b1c-acc6-a9501b0ea478-istio-envoy" pod "81095022-acb2-4b1c-acc6-a9501b0ea478" (UID: "81095022-acb2-4b1c-acc6-a9501b0ea478"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:01.613593 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.613557 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f420497-f98d-4a0c-8552-fca5c3f002f2-istio-token" pod "5f420497-f98d-4a0c-8552-fca5c3f002f2" (UID: "5f420497-f98d-4a0c-8552-fca5c3f002f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:01.701850 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.701805 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66kb5\" (UniqueName: \"kubernetes.io/projected/81095022-acb2-4b1c-acc6-a9501b0ea478-kube-api-access-66kb5\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:01.701993 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.701853 4440 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81095022-acb2-4b1c-acc6-a9501b0ea478-istio-envoy\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:01.701993 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.701882 4440 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81095022-acb2-4b1c-acc6-a9501b0ea478-istiod-ca-cert\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:01.701993 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.701897 4440 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81095022-acb2-4b1c-acc6-a9501b0ea478-istio-data\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:01.701993 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.701911 4440 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81095022-acb2-4b1c-acc6-a9501b0ea478-istio-token\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:01.701993 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.701925 4440 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f420497-f98d-4a0c-8552-fca5c3f002f2-istio-envoy\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:01.701993 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.701953 4440 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81095022-acb2-4b1c-acc6-a9501b0ea478-workload-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:01.701993 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.701966 4440 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f420497-f98d-4a0c-8552-fca5c3f002f2-istio-token\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:01.701993 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.701980 4440 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81095022-acb2-4b1c-acc6-a9501b0ea478-istio-podinfo\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:01.702248 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.701994 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsh4t\" (UniqueName: \"kubernetes.io/projected/5f420497-f98d-4a0c-8552-fca5c3f002f2-kube-api-access-tsh4t\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:01.702248 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.702009 4440 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81095022-acb2-4b1c-acc6-a9501b0ea478-credential-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:01.702248 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.702034 4440 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f420497-f98d-4a0c-8552-fca5c3f002f2-istio-podinfo\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:01.763143 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.763111 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" Oct 31 01:15:01.771405 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.771368 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3" path="/var/lib/kubelet/pods/0f5d3e96-6c6e-4b38-b3e0-ce2e3db70fa3/volumes" Oct 31 01:15:01.771962 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.771944 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b4e5ddd-9933-47e1-9265-085858232375" path="/var/lib/kubelet/pods/3b4e5ddd-9933-47e1-9265-085858232375/volumes" Oct 31 01:15:01.772371 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.772358 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f8c2432-9405-4f4f-b24f-1b73ea0da86a" path="/var/lib/kubelet/pods/8f8c2432-9405-4f4f-b24f-1b73ea0da86a/volumes" Oct 31 01:15:01.773210 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.773194 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b192b955-7018-4d48-b3c6-c58d86eeb5b2" path="/var/lib/kubelet/pods/b192b955-7018-4d48-b3c6-c58d86eeb5b2/volumes" Oct 31 01:15:01.803470 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.803081 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ee3eb69-94de-447c-b9ec-194a0458f2e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee3eb69-94de-447c-b9ec-194a0458f2e7-istio-podinfo\") pod \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\" (UID: \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\") " Oct 31 01:15:01.803470 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.803153 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ee3eb69-94de-447c-b9ec-194a0458f2e7-istio-token\" (UniqueName: \"kubernetes.io/projected/7ee3eb69-94de-447c-b9ec-194a0458f2e7-istio-token\") pod \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\" (UID: \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\") " Oct 31 01:15:01.803470 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.803290 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee3eb69-94de-447c-b9ec-194a0458f2e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee3eb69-94de-447c-b9ec-194a0458f2e7-workload-socket\") pod \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\" (UID: \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\") " Oct 31 01:15:01.803470 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.803324 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee3eb69-94de-447c-b9ec-194a0458f2e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee3eb69-94de-447c-b9ec-194a0458f2e7-credential-socket\") pod \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\" (UID: \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\") " Oct 31 01:15:01.803470 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.803355 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee3eb69-94de-447c-b9ec-194a0458f2e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee3eb69-94de-447c-b9ec-194a0458f2e7-istio-envoy\") pod \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\" (UID: \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\") " Oct 31 01:15:01.803931 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.803513 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee3eb69-94de-447c-b9ec-194a0458f2e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee3eb69-94de-447c-b9ec-194a0458f2e7-workload-certs\") pod \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\" (UID: \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\") " Oct 31 01:15:01.803931 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.803577 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ee3eb69-94de-447c-b9ec-194a0458f2e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee3eb69-94de-447c-b9ec-194a0458f2e7-istiod-ca-cert\") pod \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\" (UID: \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\") " Oct 31 01:15:01.803931 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.803619 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ee3eb69-94de-447c-b9ec-194a0458f2e7-kube-api-access-rhc6g\" (UniqueName: \"kubernetes.io/projected/7ee3eb69-94de-447c-b9ec-194a0458f2e7-kube-api-access-rhc6g\") pod \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\" (UID: \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\") " Oct 31 01:15:01.803931 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.803652 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee3eb69-94de-447c-b9ec-194a0458f2e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee3eb69-94de-447c-b9ec-194a0458f2e7-istio-data\") pod \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\" (UID: \"7ee3eb69-94de-447c-b9ec-194a0458f2e7\") " Oct 31 01:15:01.805057 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.804687 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee3eb69-94de-447c-b9ec-194a0458f2e7-credential-socket" pod "7ee3eb69-94de-447c-b9ec-194a0458f2e7" (UID: "7ee3eb69-94de-447c-b9ec-194a0458f2e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:01.805497 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.805239 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee3eb69-94de-447c-b9ec-194a0458f2e7-istio-data" pod "7ee3eb69-94de-447c-b9ec-194a0458f2e7" (UID: "7ee3eb69-94de-447c-b9ec-194a0458f2e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:01.805763 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.805742 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee3eb69-94de-447c-b9ec-194a0458f2e7-istiod-ca-cert" pod "7ee3eb69-94de-447c-b9ec-194a0458f2e7" (UID: "7ee3eb69-94de-447c-b9ec-194a0458f2e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:01.805986 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.805960 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee3eb69-94de-447c-b9ec-194a0458f2e7-workload-certs" pod "7ee3eb69-94de-447c-b9ec-194a0458f2e7" (UID: "7ee3eb69-94de-447c-b9ec-194a0458f2e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:01.806103 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.806075 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee3eb69-94de-447c-b9ec-194a0458f2e7-workload-socket" pod "7ee3eb69-94de-447c-b9ec-194a0458f2e7" (UID: "7ee3eb69-94de-447c-b9ec-194a0458f2e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:01.809147 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.809106 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee3eb69-94de-447c-b9ec-194a0458f2e7-kube-api-access-rhc6g" pod "7ee3eb69-94de-447c-b9ec-194a0458f2e7" (UID: "7ee3eb69-94de-447c-b9ec-194a0458f2e7"). InnerVolumeSpecName "kube-api-access-rhc6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:01.809533 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.809500 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee3eb69-94de-447c-b9ec-194a0458f2e7-istio-envoy" pod "7ee3eb69-94de-447c-b9ec-194a0458f2e7" (UID: "7ee3eb69-94de-447c-b9ec-194a0458f2e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:01.809920 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.809889 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ee3eb69-94de-447c-b9ec-194a0458f2e7-istio-podinfo" pod "7ee3eb69-94de-447c-b9ec-194a0458f2e7" (UID: "7ee3eb69-94de-447c-b9ec-194a0458f2e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:15:01.811253 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.811227 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee3eb69-94de-447c-b9ec-194a0458f2e7-istio-token" pod "7ee3eb69-94de-447c-b9ec-194a0458f2e7" (UID: "7ee3eb69-94de-447c-b9ec-194a0458f2e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:01.915468 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.915378 4440 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee3eb69-94de-447c-b9ec-194a0458f2e7-workload-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:01.915468 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.915441 4440 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee3eb69-94de-447c-b9ec-194a0458f2e7-credential-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:01.915468 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.915458 4440 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee3eb69-94de-447c-b9ec-194a0458f2e7-istio-envoy\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:01.915468 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.915471 4440 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee3eb69-94de-447c-b9ec-194a0458f2e7-workload-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:01.915762 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.915485 4440 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee3eb69-94de-447c-b9ec-194a0458f2e7-istiod-ca-cert\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:01.915762 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.915502 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhc6g\" (UniqueName: \"kubernetes.io/projected/7ee3eb69-94de-447c-b9ec-194a0458f2e7-kube-api-access-rhc6g\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:01.915762 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.915517 4440 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee3eb69-94de-447c-b9ec-194a0458f2e7-istio-data\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:01.915762 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.915531 4440 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee3eb69-94de-447c-b9ec-194a0458f2e7-istio-podinfo\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:01.915762 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.915544 4440 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee3eb69-94de-447c-b9ec-194a0458f2e7-istio-token\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:01.950027 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.949932 4440 generic.go:356] "Generic (PLEG): container finished" podID="877c89f5-20ff-4b40-a161-15b6e135c4b9" containerID="c9a08bb7a4f170729844228c2430295ab8460845873b72e5b7f3128266148f8f" exitCode=60 Oct 31 01:15:01.950194 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.950016 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-487a6c-29890155-gpd4t" event={"ID":"877c89f5-20ff-4b40-a161-15b6e135c4b9","Type":"ContainerDied","Data":"c9a08bb7a4f170729844228c2430295ab8460845873b72e5b7f3128266148f8f"} Oct 31 01:15:01.950320 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.950300 4440 scope.go:122] "RemoveContainer" containerID="c9a08bb7a4f170729844228c2430295ab8460845873b72e5b7f3128266148f8f" Oct 31 01:15:01.954220 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.954188 4440 generic.go:356] "Generic (PLEG): container finished" podID="81095022-acb2-4b1c-acc6-a9501b0ea478" containerID="eade63983f78f236269934cc635215faffbfcadf3990e2d1fb7b34ccf28dda43" exitCode=0 Oct 31 01:15:01.954366 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.954346 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" event={"ID":"81095022-acb2-4b1c-acc6-a9501b0ea478","Type":"ContainerDied","Data":"eade63983f78f236269934cc635215faffbfcadf3990e2d1fb7b34ccf28dda43"} Oct 31 01:15:01.954450 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.954403 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" event={"ID":"81095022-acb2-4b1c-acc6-a9501b0ea478","Type":"ContainerDied","Data":"18ac5f401f72290cf5f2c8fc8b1690dfee260038e49a7b228181d2202fef794e"} Oct 31 01:15:01.954539 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.954449 4440 scope.go:122] "RemoveContainer" containerID="eade63983f78f236269934cc635215faffbfcadf3990e2d1fb7b34ccf28dda43" Oct 31 01:15:01.954601 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.954588 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8" Oct 31 01:15:01.956798 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.956601 4440 generic.go:356] "Generic (PLEG): container finished" podID="5f420497-f98d-4a0c-8552-fca5c3f002f2" containerID="43a4bbc42c552d99a20b0ed441a4ece0bbaab3bcb4f7a2cfa71b6d4f1e76af2c" exitCode=0 Oct 31 01:15:01.956798 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.956678 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" Oct 31 01:15:01.956798 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.956710 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" event={"ID":"5f420497-f98d-4a0c-8552-fca5c3f002f2","Type":"ContainerDied","Data":"43a4bbc42c552d99a20b0ed441a4ece0bbaab3bcb4f7a2cfa71b6d4f1e76af2c"} Oct 31 01:15:01.956798 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.956758 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58" event={"ID":"5f420497-f98d-4a0c-8552-fca5c3f002f2","Type":"ContainerDied","Data":"d3e9828f5aa49a3f4215f928b605946910bfef359c08da2d5eb380358d3c8b9e"} Oct 31 01:15:01.959144 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.959117 4440 generic.go:356] "Generic (PLEG): container finished" podID="7ee3eb69-94de-447c-b9ec-194a0458f2e7" containerID="6ced7779087cd90b78a891696327fcffe5daf642c5fe82ef100d9f7576911c49" exitCode=0 Oct 31 01:15:01.959277 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.959182 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" Oct 31 01:15:01.959277 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.959211 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" event={"ID":"7ee3eb69-94de-447c-b9ec-194a0458f2e7","Type":"ContainerDied","Data":"6ced7779087cd90b78a891696327fcffe5daf642c5fe82ef100d9f7576911c49"} Oct 31 01:15:01.959277 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.959260 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw" event={"ID":"7ee3eb69-94de-447c-b9ec-194a0458f2e7","Type":"ContainerDied","Data":"76c9a5839359035a00ebea7b8bbbe1f5573ee1e3125d82987e0551ad2642c713"} Oct 31 01:15:01.966777 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.966746 4440 scope.go:122] "RemoveContainer" containerID="eade63983f78f236269934cc635215faffbfcadf3990e2d1fb7b34ccf28dda43" Oct 31 01:15:01.967540 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:15:01.967237 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eade63983f78f236269934cc635215faffbfcadf3990e2d1fb7b34ccf28dda43\": container with ID starting with eade63983f78f236269934cc635215faffbfcadf3990e2d1fb7b34ccf28dda43 not found: ID does not exist" containerID="eade63983f78f236269934cc635215faffbfcadf3990e2d1fb7b34ccf28dda43" Oct 31 01:15:01.967540 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.967301 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-kdh9z" event={"ID":"c7c55c28-c3ca-4832-9559-b05e99a3995d","Type":"ContainerStarted","Data":"9e89c47bdd9511fda363a7938422dea74098c6e015e76b73d05a97c48fd7d623"} Oct 31 01:15:01.967540 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.967283 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eade63983f78f236269934cc635215faffbfcadf3990e2d1fb7b34ccf28dda43"} err="failed to get container status \"eade63983f78f236269934cc635215faffbfcadf3990e2d1fb7b34ccf28dda43\": rpc error: code = NotFound desc = could not find container \"eade63983f78f236269934cc635215faffbfcadf3990e2d1fb7b34ccf28dda43\": container with ID starting with eade63983f78f236269934cc635215faffbfcadf3990e2d1fb7b34ccf28dda43 not found: ID does not exist" Oct 31 01:15:01.967540 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.967365 4440 scope.go:122] "RemoveContainer" containerID="43a4bbc42c552d99a20b0ed441a4ece0bbaab3bcb4f7a2cfa71b6d4f1e76af2c" Oct 31 01:15:01.984069 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.984039 4440 scope.go:122] "RemoveContainer" containerID="43a4bbc42c552d99a20b0ed441a4ece0bbaab3bcb4f7a2cfa71b6d4f1e76af2c" Oct 31 01:15:01.984396 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:15:01.984374 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43a4bbc42c552d99a20b0ed441a4ece0bbaab3bcb4f7a2cfa71b6d4f1e76af2c\": container with ID starting with 43a4bbc42c552d99a20b0ed441a4ece0bbaab3bcb4f7a2cfa71b6d4f1e76af2c not found: ID does not exist" containerID="43a4bbc42c552d99a20b0ed441a4ece0bbaab3bcb4f7a2cfa71b6d4f1e76af2c" Oct 31 01:15:01.984465 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.984407 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43a4bbc42c552d99a20b0ed441a4ece0bbaab3bcb4f7a2cfa71b6d4f1e76af2c"} err="failed to get container status \"43a4bbc42c552d99a20b0ed441a4ece0bbaab3bcb4f7a2cfa71b6d4f1e76af2c\": rpc error: code = NotFound desc = could not find container \"43a4bbc42c552d99a20b0ed441a4ece0bbaab3bcb4f7a2cfa71b6d4f1e76af2c\": container with ID starting with 43a4bbc42c552d99a20b0ed441a4ece0bbaab3bcb4f7a2cfa71b6d4f1e76af2c not found: ID does not exist" Oct 31 01:15:01.984465 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.984443 4440 scope.go:122] "RemoveContainer" containerID="6ced7779087cd90b78a891696327fcffe5daf642c5fe82ef100d9f7576911c49" Oct 31 01:15:01.991333 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.991276 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-kdh9z" podStartSLOduration=1.991258661 podStartE2EDuration="1.991258661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:01.986586496 +0000 UTC m=+1146.345308813" watchObservedRunningTime="2026-10-31 01:15:01.991258661 +0000 UTC m=+1146.349980976" Oct 31 01:15:01.996409 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.996363 4440 scope.go:122] "RemoveContainer" containerID="6ced7779087cd90b78a891696327fcffe5daf642c5fe82ef100d9f7576911c49" Oct 31 01:15:01.997016 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:15:01.996821 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ced7779087cd90b78a891696327fcffe5daf642c5fe82ef100d9f7576911c49\": container with ID starting with 6ced7779087cd90b78a891696327fcffe5daf642c5fe82ef100d9f7576911c49 not found: ID does not exist" containerID="6ced7779087cd90b78a891696327fcffe5daf642c5fe82ef100d9f7576911c49" Oct 31 01:15:01.997016 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:01.996860 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ced7779087cd90b78a891696327fcffe5daf642c5fe82ef100d9f7576911c49"} err="failed to get container status \"6ced7779087cd90b78a891696327fcffe5daf642c5fe82ef100d9f7576911c49\": rpc error: code = NotFound desc = could not find container \"6ced7779087cd90b78a891696327fcffe5daf642c5fe82ef100d9f7576911c49\": container with ID starting with 6ced7779087cd90b78a891696327fcffe5daf642c5fe82ef100d9f7576911c49 not found: ID does not exist" Oct 31 01:15:02.007315 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:02.007280 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8"] Oct 31 01:15:02.013082 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:02.013037 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-d5lq8"] Oct 31 01:15:02.032934 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:02.032896 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58"] Oct 31 01:15:02.036987 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:02.036953 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-j6w58"] Oct 31 01:15:02.057297 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:02.057255 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw"] Oct 31 01:15:02.072238 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:02.072199 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-28ff08-openshift-default-7c95bbc97f-fzhfw"] Oct 31 01:15:02.178219 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:02.178182 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:02.326309 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:02.326273 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:02.504832 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:02.504737 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:02.628314 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:02.628274 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:02.888956 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:02.888850 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:02.977826 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:02.977795 4440 generic.go:356] "Generic (PLEG): container finished" podID="877c89f5-20ff-4b40-a161-15b6e135c4b9" containerID="bf08e676cb9ef26c7c0c9f4f1b9c3a35f6a3f7d064e8cce5da6e499b57a13d4a" exitCode=60 Oct 31 01:15:02.978033 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:02.977875 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-487a6c-29890155-gpd4t" event={"ID":"877c89f5-20ff-4b40-a161-15b6e135c4b9","Type":"ContainerDied","Data":"bf08e676cb9ef26c7c0c9f4f1b9c3a35f6a3f7d064e8cce5da6e499b57a13d4a"} Oct 31 01:15:02.978033 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:02.977936 4440 scope.go:122] "RemoveContainer" containerID="c9a08bb7a4f170729844228c2430295ab8460845873b72e5b7f3128266148f8f" Oct 31 01:15:02.978285 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:02.978268 4440 scope.go:122] "RemoveContainer" containerID="bf08e676cb9ef26c7c0c9f4f1b9c3a35f6a3f7d064e8cce5da6e499b57a13d4a" Oct 31 01:15:02.978640 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:15:02.978619 4440 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-487a6c-29890155-gpd4t_odh-ai-gateway-infra(877c89f5-20ff-4b40-a161-15b6e135c4b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-487a6c-29890155-gpd4t" podUID="877c89f5-20ff-4b40-a161-15b6e135c4b9" Oct 31 01:15:02.982967 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:02.982943 4440 generic.go:356] "Generic (PLEG): container finished" podID="c7c55c28-c3ca-4832-9559-b05e99a3995d" containerID="9e89c47bdd9511fda363a7938422dea74098c6e015e76b73d05a97c48fd7d623" exitCode=60 Oct 31 01:15:02.983096 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:02.983026 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-kdh9z" event={"ID":"c7c55c28-c3ca-4832-9559-b05e99a3995d","Type":"ContainerDied","Data":"9e89c47bdd9511fda363a7938422dea74098c6e015e76b73d05a97c48fd7d623"} Oct 31 01:15:02.983286 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:02.983270 4440 scope.go:122] "RemoveContainer" containerID="9e89c47bdd9511fda363a7938422dea74098c6e015e76b73d05a97c48fd7d623" Oct 31 01:15:03.455473 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:03.455413 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg"] Oct 31 01:15:03.529041 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:03.528945 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:03.671371 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:03.671334 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:03.771772 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:03.771739 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f420497-f98d-4a0c-8552-fca5c3f002f2" path="/var/lib/kubelet/pods/5f420497-f98d-4a0c-8552-fca5c3f002f2/volumes" Oct 31 01:15:03.772204 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:03.772191 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ee3eb69-94de-447c-b9ec-194a0458f2e7" path="/var/lib/kubelet/pods/7ee3eb69-94de-447c-b9ec-194a0458f2e7/volumes" Oct 31 01:15:03.772658 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:03.772642 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81095022-acb2-4b1c-acc6-a9501b0ea478" path="/var/lib/kubelet/pods/81095022-acb2-4b1c-acc6-a9501b0ea478/volumes" Oct 31 01:15:04.000905 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:04.000872 4440 scope.go:122] "RemoveContainer" containerID="bf08e676cb9ef26c7c0c9f4f1b9c3a35f6a3f7d064e8cce5da6e499b57a13d4a" Oct 31 01:15:04.001213 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:15:04.001189 4440 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-487a6c-29890155-gpd4t_odh-ai-gateway-infra(877c89f5-20ff-4b40-a161-15b6e135c4b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-487a6c-29890155-gpd4t" podUID="877c89f5-20ff-4b40-a161-15b6e135c4b9" Oct 31 01:15:04.002868 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:04.002839 4440 generic.go:356] "Generic (PLEG): container finished" podID="c7c55c28-c3ca-4832-9559-b05e99a3995d" containerID="7e2dd1609cc0bb326e96183017f39a031064731ca3c9b553784ed47a3bb212cb" exitCode=60 Oct 31 01:15:04.002963 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:04.002919 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-kdh9z" event={"ID":"c7c55c28-c3ca-4832-9559-b05e99a3995d","Type":"ContainerDied","Data":"7e2dd1609cc0bb326e96183017f39a031064731ca3c9b553784ed47a3bb212cb"} Oct 31 01:15:04.003045 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:04.003031 4440 scope.go:122] "RemoveContainer" containerID="9e89c47bdd9511fda363a7938422dea74098c6e015e76b73d05a97c48fd7d623" Oct 31 01:15:04.003190 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:04.003139 4440 scope.go:122] "RemoveContainer" containerID="7e2dd1609cc0bb326e96183017f39a031064731ca3c9b553784ed47a3bb212cb" Oct 31 01:15:04.003587 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:15:04.003564 4440 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890155-kdh9z_odh-ai-gateway-infra(c7c55c28-c3ca-4832-9559-b05e99a3995d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-kdh9z" podUID="c7c55c28-c3ca-4832-9559-b05e99a3995d" Oct 31 01:15:04.347650 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:04.347601 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:04.553468 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:04.549126 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:04.663856 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:04.663736 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:04.775161 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:04.775122 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:05.018019 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:05.017984 4440 scope.go:122] "RemoveContainer" containerID="7e2dd1609cc0bb326e96183017f39a031064731ca3c9b553784ed47a3bb212cb" Oct 31 01:15:05.018378 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:15:05.018350 4440 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890155-kdh9z_odh-ai-gateway-infra(c7c55c28-c3ca-4832-9559-b05e99a3995d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-kdh9z" podUID="c7c55c28-c3ca-4832-9559-b05e99a3995d" Oct 31 01:15:05.172593 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:05.172548 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:05.271791 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:05.271696 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:05.894287 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:05.894247 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:06.370047 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:06.370015 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-acdaed-55478d5599-md22s" podUID="e4160358-c655-4a6b-bb39-fc8677804241" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:15:06.497130 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:06.497088 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:06.733321 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:06.733263 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:06.853032 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:06.852966 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl"] Oct 31 01:15:06.853638 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:06.853599 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" podUID="cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf" containerName="istio-proxy" containerID="cri-o://eee38b000e0b1934cca38ee3960c9cf056fac56e9bea8e3dde35d32f831b7406" gracePeriod=30 Oct 31 01:15:06.857467 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:06.857434 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw"] Oct 31 01:15:06.861863 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:06.861828 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9"] Oct 31 01:15:06.862051 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:06.862025 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" podUID="117032f9-b742-432f-ac5f-70c9982b919f" containerName="istio-proxy" containerID="cri-o://47d075176e3198c8e4bdd83ab0bdc4c02916e8a58f7aa8ecbe149e38243074af" gracePeriod=30 Oct 31 01:15:06.878495 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:06.878455 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:07.011690 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:07.011590 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:07.049955 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:07.049877 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" podUID="823db35e-e37c-448d-a70d-96b5a218ac5e" containerName="istio-proxy" containerID="cri-o://305f28bdc13501bf40eb78a29b4b962936182bc5ac8c415e90ded874637b45d5" gracePeriod=30 Oct 31 01:15:07.566276 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:07.566229 4440 patch_prober.go:29] interesting pod/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:15:07.566468 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:07.566307 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" podUID="117032f9-b742-432f-ac5f-70c9982b919f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:15:07.630001 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:07.629967 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:08.310056 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:08.310017 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:08.804450 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:08.804399 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-487a6c-6f784564bc-c6qb5"] Oct 31 01:15:08.903084 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:08.903031 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:09.023801 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:09.023755 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:09.153531 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:09.144258 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:10.885939 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:10.885900 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:11.026999 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:11.026958 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:11.543481 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:11.543452 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-acdaed-55478d5599-md22s" Oct 31 01:15:11.669316 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:11.669275 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4160358-c655-4a6b-bb39-fc8677804241-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4160358-c655-4a6b-bb39-fc8677804241-plugins-config-volume\") pod \"e4160358-c655-4a6b-bb39-fc8677804241\" (UID: \"e4160358-c655-4a6b-bb39-fc8677804241\") " Oct 31 01:15:11.669529 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:11.669444 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4160358-c655-4a6b-bb39-fc8677804241-kube-api-access-hfqjb\" (UniqueName: \"kubernetes.io/projected/e4160358-c655-4a6b-bb39-fc8677804241-kube-api-access-hfqjb\") pod \"e4160358-c655-4a6b-bb39-fc8677804241\" (UID: \"e4160358-c655-4a6b-bb39-fc8677804241\") " Oct 31 01:15:11.669697 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:11.669674 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4160358-c655-4a6b-bb39-fc8677804241-plugins-config-volume" pod "e4160358-c655-4a6b-bb39-fc8677804241" (UID: "e4160358-c655-4a6b-bb39-fc8677804241"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:11.669842 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:11.669820 4440 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4160358-c655-4a6b-bb39-fc8677804241-plugins-config-volume\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:11.672081 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:11.672044 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4160358-c655-4a6b-bb39-fc8677804241-kube-api-access-hfqjb" pod "e4160358-c655-4a6b-bb39-fc8677804241" (UID: "e4160358-c655-4a6b-bb39-fc8677804241"). InnerVolumeSpecName "kube-api-access-hfqjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:11.770660 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:11.770579 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfqjb\" (UniqueName: \"kubernetes.io/projected/e4160358-c655-4a6b-bb39-fc8677804241-kube-api-access-hfqjb\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:12.101239 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.101199 4440 generic.go:356] "Generic (PLEG): container finished" podID="117032f9-b742-432f-ac5f-70c9982b919f" containerID="47d075176e3198c8e4bdd83ab0bdc4c02916e8a58f7aa8ecbe149e38243074af" exitCode=0 Oct 31 01:15:12.101730 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.101547 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" event={"ID":"117032f9-b742-432f-ac5f-70c9982b919f","Type":"ContainerDied","Data":"47d075176e3198c8e4bdd83ab0bdc4c02916e8a58f7aa8ecbe149e38243074af"} Oct 31 01:15:12.104764 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.104726 4440 generic.go:356] "Generic (PLEG): container finished" podID="e4160358-c655-4a6b-bb39-fc8677804241" containerID="30fcda6740c24446c9a353da5eaecabbf6486015e8b17082862fbd1e1421fca2" exitCode=0 Oct 31 01:15:12.104944 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.104925 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-acdaed-55478d5599-md22s" event={"ID":"e4160358-c655-4a6b-bb39-fc8677804241","Type":"ContainerDied","Data":"30fcda6740c24446c9a353da5eaecabbf6486015e8b17082862fbd1e1421fca2"} Oct 31 01:15:12.105014 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.104980 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-acdaed-55478d5599-md22s" event={"ID":"e4160358-c655-4a6b-bb39-fc8677804241","Type":"ContainerDied","Data":"bc6f9059d5390a5f1f07241e8769950da7afb9c6aeaccab20b2edc31aa17b3d7"} Oct 31 01:15:12.105072 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.105030 4440 scope.go:122] "RemoveContainer" containerID="30fcda6740c24446c9a353da5eaecabbf6486015e8b17082862fbd1e1421fca2" Oct 31 01:15:12.105195 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.105181 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-acdaed-55478d5599-md22s" Oct 31 01:15:12.120475 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.109873 4440 generic.go:356] "Generic (PLEG): container finished" podID="cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf" containerID="eee38b000e0b1934cca38ee3960c9cf056fac56e9bea8e3dde35d32f831b7406" exitCode=0 Oct 31 01:15:12.120475 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.109974 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" event={"ID":"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf","Type":"ContainerDied","Data":"eee38b000e0b1934cca38ee3960c9cf056fac56e9bea8e3dde35d32f831b7406"} Oct 31 01:15:12.161764 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.161730 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-acdaed-55478d5599-md22s"] Oct 31 01:15:12.167411 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.167374 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" Oct 31 01:15:12.167961 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.167753 4440 scope.go:122] "RemoveContainer" containerID="30fcda6740c24446c9a353da5eaecabbf6486015e8b17082862fbd1e1421fca2" Oct 31 01:15:12.168169 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:15:12.168141 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30fcda6740c24446c9a353da5eaecabbf6486015e8b17082862fbd1e1421fca2\": container with ID starting with 30fcda6740c24446c9a353da5eaecabbf6486015e8b17082862fbd1e1421fca2 not found: ID does not exist" containerID="30fcda6740c24446c9a353da5eaecabbf6486015e8b17082862fbd1e1421fca2" Oct 31 01:15:12.168254 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.168182 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30fcda6740c24446c9a353da5eaecabbf6486015e8b17082862fbd1e1421fca2"} err="failed to get container status \"30fcda6740c24446c9a353da5eaecabbf6486015e8b17082862fbd1e1421fca2\": rpc error: code = NotFound desc = could not find container \"30fcda6740c24446c9a353da5eaecabbf6486015e8b17082862fbd1e1421fca2\": container with ID starting with 30fcda6740c24446c9a353da5eaecabbf6486015e8b17082862fbd1e1421fca2 not found: ID does not exist" Oct 31 01:15:12.168254 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.168215 4440 scope.go:122] "RemoveContainer" containerID="c26e3bab275a94d05499b10f9528f1f51aa858a8dba45c8ba18710f2b1c4811a" Oct 31 01:15:12.180701 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.180666 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-acdaed-55478d5599-md22s"] Oct 31 01:15:12.181430 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.181389 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" Oct 31 01:15:12.278081 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.277809 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/117032f9-b742-432f-ac5f-70c9982b919f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/117032f9-b742-432f-ac5f-70c9982b919f-credential-socket\") pod \"117032f9-b742-432f-ac5f-70c9982b919f\" (UID: \"117032f9-b742-432f-ac5f-70c9982b919f\") " Oct 31 01:15:12.278081 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.277858 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/117032f9-b742-432f-ac5f-70c9982b919f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/117032f9-b742-432f-ac5f-70c9982b919f-workload-socket\") pod \"117032f9-b742-432f-ac5f-70c9982b919f\" (UID: \"117032f9-b742-432f-ac5f-70c9982b919f\") " Oct 31 01:15:12.278081 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.277888 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/117032f9-b742-432f-ac5f-70c9982b919f-kube-api-access-v8j59\" (UniqueName: \"kubernetes.io/projected/117032f9-b742-432f-ac5f-70c9982b919f-kube-api-access-v8j59\") pod \"117032f9-b742-432f-ac5f-70c9982b919f\" (UID: \"117032f9-b742-432f-ac5f-70c9982b919f\") " Oct 31 01:15:12.278081 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.277924 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/117032f9-b742-432f-ac5f-70c9982b919f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/117032f9-b742-432f-ac5f-70c9982b919f-istio-envoy\") pod \"117032f9-b742-432f-ac5f-70c9982b919f\" (UID: \"117032f9-b742-432f-ac5f-70c9982b919f\") " Oct 31 01:15:12.278081 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.277964 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-istio-envoy\") pod \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\" (UID: \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\") " Oct 31 01:15:12.278081 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.278029 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-workload-certs\") pod \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\" (UID: \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\") " Oct 31 01:15:12.278081 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.278079 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-istio-data\") pod \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\" (UID: \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\") " Oct 31 01:15:12.278670 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.278107 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-istio-podinfo\") pod \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\" (UID: \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\") " Oct 31 01:15:12.278670 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.278139 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/117032f9-b742-432f-ac5f-70c9982b919f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/117032f9-b742-432f-ac5f-70c9982b919f-istio-data\") pod \"117032f9-b742-432f-ac5f-70c9982b919f\" (UID: \"117032f9-b742-432f-ac5f-70c9982b919f\") " Oct 31 01:15:12.278670 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.278207 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-istio-token\" (UniqueName: \"kubernetes.io/projected/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-istio-token\") pod \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\" (UID: \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\") " Oct 31 01:15:12.278670 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.278255 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/117032f9-b742-432f-ac5f-70c9982b919f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/117032f9-b742-432f-ac5f-70c9982b919f-workload-certs\") pod \"117032f9-b742-432f-ac5f-70c9982b919f\" (UID: \"117032f9-b742-432f-ac5f-70c9982b919f\") " Oct 31 01:15:12.278670 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.278312 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/117032f9-b742-432f-ac5f-70c9982b919f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/117032f9-b742-432f-ac5f-70c9982b919f-istio-podinfo\") pod \"117032f9-b742-432f-ac5f-70c9982b919f\" (UID: \"117032f9-b742-432f-ac5f-70c9982b919f\") " Oct 31 01:15:12.278670 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.278362 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/117032f9-b742-432f-ac5f-70c9982b919f-istio-token\" (UniqueName: \"kubernetes.io/projected/117032f9-b742-432f-ac5f-70c9982b919f-istio-token\") pod \"117032f9-b742-432f-ac5f-70c9982b919f\" (UID: \"117032f9-b742-432f-ac5f-70c9982b919f\") " Oct 31 01:15:12.278670 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.278407 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-kube-api-access-5qtx2\" (UniqueName: \"kubernetes.io/projected/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-kube-api-access-5qtx2\") pod \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\" (UID: \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\") " Oct 31 01:15:12.278670 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.278450 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/117032f9-b742-432f-ac5f-70c9982b919f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/117032f9-b742-432f-ac5f-70c9982b919f-istiod-ca-cert\") pod \"117032f9-b742-432f-ac5f-70c9982b919f\" (UID: \"117032f9-b742-432f-ac5f-70c9982b919f\") " Oct 31 01:15:12.278670 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.278493 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-istiod-ca-cert\") pod \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\" (UID: \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\") " Oct 31 01:15:12.278670 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.278519 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-credential-socket\") pod \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\" (UID: \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\") " Oct 31 01:15:12.278670 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.278556 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-workload-socket\") pod \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\" (UID: \"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf\") " Oct 31 01:15:12.279492 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.279197 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-workload-socket" pod "cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf" (UID: "cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:12.279492 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.279440 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/117032f9-b742-432f-ac5f-70c9982b919f-credential-socket" pod "117032f9-b742-432f-ac5f-70c9982b919f" (UID: "117032f9-b742-432f-ac5f-70c9982b919f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:12.279684 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.279619 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/117032f9-b742-432f-ac5f-70c9982b919f-workload-socket" pod "117032f9-b742-432f-ac5f-70c9982b919f" (UID: "117032f9-b742-432f-ac5f-70c9982b919f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:12.280537 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.280506 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-workload-certs" pod "cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf" (UID: "cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:12.281495 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.281073 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-credential-socket" pod "cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf" (UID: "cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:12.281495 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.281260 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-istio-data" pod "cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf" (UID: "cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:12.281495 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.281319 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/117032f9-b742-432f-ac5f-70c9982b919f-workload-certs" pod "117032f9-b742-432f-ac5f-70c9982b919f" (UID: "117032f9-b742-432f-ac5f-70c9982b919f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:12.281495 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.281400 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/117032f9-b742-432f-ac5f-70c9982b919f-istio-data" pod "117032f9-b742-432f-ac5f-70c9982b919f" (UID: "117032f9-b742-432f-ac5f-70c9982b919f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:12.282828 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.282552 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/117032f9-b742-432f-ac5f-70c9982b919f-istiod-ca-cert" pod "117032f9-b742-432f-ac5f-70c9982b919f" (UID: "117032f9-b742-432f-ac5f-70c9982b919f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:12.282828 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.282602 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-istiod-ca-cert" pod "cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf" (UID: "cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:12.284442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.284327 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/117032f9-b742-432f-ac5f-70c9982b919f-istio-envoy" pod "117032f9-b742-432f-ac5f-70c9982b919f" (UID: "117032f9-b742-432f-ac5f-70c9982b919f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:12.284442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.284392 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-istio-envoy" pod "cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf" (UID: "cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:12.285607 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.285579 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-kube-api-access-5qtx2" pod "cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf" (UID: "cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf"). InnerVolumeSpecName "kube-api-access-5qtx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:12.286083 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.286036 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/117032f9-b742-432f-ac5f-70c9982b919f-istio-podinfo" pod "117032f9-b742-432f-ac5f-70c9982b919f" (UID: "117032f9-b742-432f-ac5f-70c9982b919f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:15:12.286178 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.286133 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-istio-token" pod "cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf" (UID: "cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:12.286296 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.286273 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/117032f9-b742-432f-ac5f-70c9982b919f-kube-api-access-v8j59" pod "117032f9-b742-432f-ac5f-70c9982b919f" (UID: "117032f9-b742-432f-ac5f-70c9982b919f"). InnerVolumeSpecName "kube-api-access-v8j59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:12.287122 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.287096 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-istio-podinfo" pod "cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf" (UID: "cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:15:12.287910 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.287887 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/117032f9-b742-432f-ac5f-70c9982b919f-istio-token" pod "117032f9-b742-432f-ac5f-70c9982b919f" (UID: "117032f9-b742-432f-ac5f-70c9982b919f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:12.370411 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.370382 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" Oct 31 01:15:12.379874 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.379840 4440 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-istio-data\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:12.379874 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.379873 4440 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-istio-podinfo\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:12.380054 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.379884 4440 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/117032f9-b742-432f-ac5f-70c9982b919f-istio-data\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:12.380054 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.379893 4440 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-istio-token\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:12.380054 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.379904 4440 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/117032f9-b742-432f-ac5f-70c9982b919f-workload-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:12.380054 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.379912 4440 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/117032f9-b742-432f-ac5f-70c9982b919f-istio-podinfo\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:12.380054 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.379922 4440 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/117032f9-b742-432f-ac5f-70c9982b919f-istio-token\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:12.380054 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.379930 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qtx2\" (UniqueName: \"kubernetes.io/projected/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-kube-api-access-5qtx2\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:12.380054 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.379942 4440 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/117032f9-b742-432f-ac5f-70c9982b919f-istiod-ca-cert\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:12.380054 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.379950 4440 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-istiod-ca-cert\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:12.380054 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.379959 4440 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-credential-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:12.380054 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.379967 4440 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-workload-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:12.380054 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.379975 4440 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/117032f9-b742-432f-ac5f-70c9982b919f-credential-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:12.380054 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.379983 4440 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/117032f9-b742-432f-ac5f-70c9982b919f-workload-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:12.380054 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.379992 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8j59\" (UniqueName: \"kubernetes.io/projected/117032f9-b742-432f-ac5f-70c9982b919f-kube-api-access-v8j59\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:12.380054 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.380001 4440 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/117032f9-b742-432f-ac5f-70c9982b919f-istio-envoy\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:12.380054 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.380010 4440 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-istio-envoy\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:12.380054 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.380019 4440 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf-workload-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:12.481289 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.481249 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823db35e-e37c-448d-a70d-96b5a218ac5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/823db35e-e37c-448d-a70d-96b5a218ac5e-istio-envoy\") pod \"823db35e-e37c-448d-a70d-96b5a218ac5e\" (UID: \"823db35e-e37c-448d-a70d-96b5a218ac5e\") " Oct 31 01:15:12.481508 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.481315 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/823db35e-e37c-448d-a70d-96b5a218ac5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/823db35e-e37c-448d-a70d-96b5a218ac5e-istiod-ca-cert\") pod \"823db35e-e37c-448d-a70d-96b5a218ac5e\" (UID: \"823db35e-e37c-448d-a70d-96b5a218ac5e\") " Oct 31 01:15:12.481508 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.481365 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823db35e-e37c-448d-a70d-96b5a218ac5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/823db35e-e37c-448d-a70d-96b5a218ac5e-workload-certs\") pod \"823db35e-e37c-448d-a70d-96b5a218ac5e\" (UID: \"823db35e-e37c-448d-a70d-96b5a218ac5e\") " Oct 31 01:15:12.481508 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.481433 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823db35e-e37c-448d-a70d-96b5a218ac5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/823db35e-e37c-448d-a70d-96b5a218ac5e-workload-socket\") pod \"823db35e-e37c-448d-a70d-96b5a218ac5e\" (UID: \"823db35e-e37c-448d-a70d-96b5a218ac5e\") " Oct 31 01:15:12.481508 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.481491 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823db35e-e37c-448d-a70d-96b5a218ac5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/823db35e-e37c-448d-a70d-96b5a218ac5e-istio-data\") pod \"823db35e-e37c-448d-a70d-96b5a218ac5e\" (UID: \"823db35e-e37c-448d-a70d-96b5a218ac5e\") " Oct 31 01:15:12.481747 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.481531 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/823db35e-e37c-448d-a70d-96b5a218ac5e-kube-api-access-6qmrf\" (UniqueName: \"kubernetes.io/projected/823db35e-e37c-448d-a70d-96b5a218ac5e-kube-api-access-6qmrf\") pod \"823db35e-e37c-448d-a70d-96b5a218ac5e\" (UID: \"823db35e-e37c-448d-a70d-96b5a218ac5e\") " Oct 31 01:15:12.481747 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.481561 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/823db35e-e37c-448d-a70d-96b5a218ac5e-istio-token\" (UniqueName: \"kubernetes.io/projected/823db35e-e37c-448d-a70d-96b5a218ac5e-istio-token\") pod \"823db35e-e37c-448d-a70d-96b5a218ac5e\" (UID: \"823db35e-e37c-448d-a70d-96b5a218ac5e\") " Oct 31 01:15:12.481747 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.481648 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/823db35e-e37c-448d-a70d-96b5a218ac5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/823db35e-e37c-448d-a70d-96b5a218ac5e-istio-podinfo\") pod \"823db35e-e37c-448d-a70d-96b5a218ac5e\" (UID: \"823db35e-e37c-448d-a70d-96b5a218ac5e\") " Oct 31 01:15:12.481747 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.481678 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823db35e-e37c-448d-a70d-96b5a218ac5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/823db35e-e37c-448d-a70d-96b5a218ac5e-credential-socket\") pod \"823db35e-e37c-448d-a70d-96b5a218ac5e\" (UID: \"823db35e-e37c-448d-a70d-96b5a218ac5e\") " Oct 31 01:15:12.481747 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.481702 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823db35e-e37c-448d-a70d-96b5a218ac5e-workload-socket" pod "823db35e-e37c-448d-a70d-96b5a218ac5e" (UID: "823db35e-e37c-448d-a70d-96b5a218ac5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:12.481747 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.481720 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823db35e-e37c-448d-a70d-96b5a218ac5e-workload-certs" pod "823db35e-e37c-448d-a70d-96b5a218ac5e" (UID: "823db35e-e37c-448d-a70d-96b5a218ac5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:12.481747 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.481744 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/823db35e-e37c-448d-a70d-96b5a218ac5e-istiod-ca-cert" pod "823db35e-e37c-448d-a70d-96b5a218ac5e" (UID: "823db35e-e37c-448d-a70d-96b5a218ac5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:12.482093 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.481955 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823db35e-e37c-448d-a70d-96b5a218ac5e-credential-socket" pod "823db35e-e37c-448d-a70d-96b5a218ac5e" (UID: "823db35e-e37c-448d-a70d-96b5a218ac5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:12.482395 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.482365 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823db35e-e37c-448d-a70d-96b5a218ac5e-istio-data" pod "823db35e-e37c-448d-a70d-96b5a218ac5e" (UID: "823db35e-e37c-448d-a70d-96b5a218ac5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:12.482538 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.482397 4440 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/823db35e-e37c-448d-a70d-96b5a218ac5e-credential-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:12.482538 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.482436 4440 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/823db35e-e37c-448d-a70d-96b5a218ac5e-istiod-ca-cert\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:12.482538 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.482450 4440 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/823db35e-e37c-448d-a70d-96b5a218ac5e-workload-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:12.482538 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.482467 4440 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/823db35e-e37c-448d-a70d-96b5a218ac5e-workload-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:12.483929 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.483904 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/823db35e-e37c-448d-a70d-96b5a218ac5e-kube-api-access-6qmrf" pod "823db35e-e37c-448d-a70d-96b5a218ac5e" (UID: "823db35e-e37c-448d-a70d-96b5a218ac5e"). InnerVolumeSpecName "kube-api-access-6qmrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:12.484018 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.483953 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823db35e-e37c-448d-a70d-96b5a218ac5e-istio-envoy" pod "823db35e-e37c-448d-a70d-96b5a218ac5e" (UID: "823db35e-e37c-448d-a70d-96b5a218ac5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:12.484018 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.483987 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/823db35e-e37c-448d-a70d-96b5a218ac5e-istio-token" pod "823db35e-e37c-448d-a70d-96b5a218ac5e" (UID: "823db35e-e37c-448d-a70d-96b5a218ac5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:12.484378 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.484357 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/823db35e-e37c-448d-a70d-96b5a218ac5e-istio-podinfo" pod "823db35e-e37c-448d-a70d-96b5a218ac5e" (UID: "823db35e-e37c-448d-a70d-96b5a218ac5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:15:12.584058 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.584018 4440 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/823db35e-e37c-448d-a70d-96b5a218ac5e-istio-data\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:12.584058 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.584053 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qmrf\" (UniqueName: \"kubernetes.io/projected/823db35e-e37c-448d-a70d-96b5a218ac5e-kube-api-access-6qmrf\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:12.584058 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.584064 4440 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/823db35e-e37c-448d-a70d-96b5a218ac5e-istio-token\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:12.584303 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.584072 4440 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/823db35e-e37c-448d-a70d-96b5a218ac5e-istio-podinfo\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:12.584303 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:12.584081 4440 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/823db35e-e37c-448d-a70d-96b5a218ac5e-istio-envoy\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:13.117622 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:13.117586 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" Oct 31 01:15:13.118105 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:13.117618 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9" event={"ID":"117032f9-b742-432f-ac5f-70c9982b919f","Type":"ContainerDied","Data":"7f2a94b35e2c86ba3b8a4b22d1c242cacd7ecd2a78d02b8a3a5b9440b9e17027"} Oct 31 01:15:13.118105 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:13.117687 4440 scope.go:122] "RemoveContainer" containerID="47d075176e3198c8e4bdd83ab0bdc4c02916e8a58f7aa8ecbe149e38243074af" Oct 31 01:15:13.120372 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:13.120349 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" Oct 31 01:15:13.120555 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:13.120354 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl" event={"ID":"cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf","Type":"ContainerDied","Data":"cf2b60db433d0806c15973332161e6c77219d1272722cc8983ad42d6a5e4c35e"} Oct 31 01:15:13.121978 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:13.121953 4440 generic.go:356] "Generic (PLEG): container finished" podID="823db35e-e37c-448d-a70d-96b5a218ac5e" containerID="305f28bdc13501bf40eb78a29b4b962936182bc5ac8c415e90ded874637b45d5" exitCode=0 Oct 31 01:15:13.122087 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:13.122003 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" Oct 31 01:15:13.122087 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:13.122036 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" event={"ID":"823db35e-e37c-448d-a70d-96b5a218ac5e","Type":"ContainerDied","Data":"305f28bdc13501bf40eb78a29b4b962936182bc5ac8c415e90ded874637b45d5"} Oct 31 01:15:13.122203 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:13.122096 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw" event={"ID":"823db35e-e37c-448d-a70d-96b5a218ac5e","Type":"ContainerDied","Data":"a190e0bbcfb64b6718490c35f249d351c116975a2d4774c2994bb4fe8a93651a"} Oct 31 01:15:13.130492 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:13.130468 4440 scope.go:122] "RemoveContainer" containerID="eee38b000e0b1934cca38ee3960c9cf056fac56e9bea8e3dde35d32f831b7406" Oct 31 01:15:13.144290 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:13.144253 4440 scope.go:122] "RemoveContainer" containerID="305f28bdc13501bf40eb78a29b4b962936182bc5ac8c415e90ded874637b45d5" Oct 31 01:15:13.154134 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:13.154108 4440 scope.go:122] "RemoveContainer" containerID="305f28bdc13501bf40eb78a29b4b962936182bc5ac8c415e90ded874637b45d5" Oct 31 01:15:13.154461 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:15:13.154412 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"305f28bdc13501bf40eb78a29b4b962936182bc5ac8c415e90ded874637b45d5\": container with ID starting with 305f28bdc13501bf40eb78a29b4b962936182bc5ac8c415e90ded874637b45d5 not found: ID does not exist" containerID="305f28bdc13501bf40eb78a29b4b962936182bc5ac8c415e90ded874637b45d5" Oct 31 01:15:13.154536 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:13.154474 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"305f28bdc13501bf40eb78a29b4b962936182bc5ac8c415e90ded874637b45d5"} err="failed to get container status \"305f28bdc13501bf40eb78a29b4b962936182bc5ac8c415e90ded874637b45d5\": rpc error: code = NotFound desc = could not find container \"305f28bdc13501bf40eb78a29b4b962936182bc5ac8c415e90ded874637b45d5\": container with ID starting with 305f28bdc13501bf40eb78a29b4b962936182bc5ac8c415e90ded874637b45d5 not found: ID does not exist" Oct 31 01:15:13.161941 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:13.161908 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw"] Oct 31 01:15:13.167203 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:13.167181 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-58pxw"] Oct 31 01:15:13.178109 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:13.178082 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl"] Oct 31 01:15:13.184233 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:13.184206 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-2x7vl"] Oct 31 01:15:13.193721 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:13.193695 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9"] Oct 31 01:15:13.205198 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:13.205160 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-acdaed-openshift-default-fd4ccfb57-v9dl9"] Oct 31 01:15:13.764131 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:13.764089 4440 scope.go:122] "RemoveContainer" containerID="bf08e676cb9ef26c7c0c9f4f1b9c3a35f6a3f7d064e8cce5da6e499b57a13d4a" Oct 31 01:15:13.772849 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:13.772817 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="117032f9-b742-432f-ac5f-70c9982b919f" path="/var/lib/kubelet/pods/117032f9-b742-432f-ac5f-70c9982b919f/volumes" Oct 31 01:15:13.773368 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:13.773346 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="823db35e-e37c-448d-a70d-96b5a218ac5e" path="/var/lib/kubelet/pods/823db35e-e37c-448d-a70d-96b5a218ac5e/volumes" Oct 31 01:15:13.773961 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:13.773935 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf" path="/var/lib/kubelet/pods/cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf/volumes" Oct 31 01:15:13.774834 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:13.774810 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4160358-c655-4a6b-bb39-fc8677804241" path="/var/lib/kubelet/pods/e4160358-c655-4a6b-bb39-fc8677804241/volumes" Oct 31 01:15:13.805654 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:13.805603 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-487a6c-6f784564bc-c6qb5" podUID="0a3e8a39-48ca-4063-93b8-8afa2f1f18ac" containerName="payload-processing" containerID="cri-o://03b0b7a38b56730c846fed5a5f497ad2d1a8b593d453e4e09eb61e8e5182d098" gracePeriod=25 Oct 31 01:15:14.133165 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:14.133126 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-487a6c-29890155-gpd4t" event={"ID":"877c89f5-20ff-4b40-a161-15b6e135c4b9","Type":"ContainerStarted","Data":"b326cc73957bde163c0d4d39d56898cbc1b701f80d5a611f3d6aa0fb4ad08aa4"} Oct 31 01:15:14.150994 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:14.150933 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-487a6c-29890155-gpd4t" podStartSLOduration=14.150915772 podStartE2EDuration="14.150915772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:14.148246726 +0000 UTC m=+1158.506969043" watchObservedRunningTime="2026-10-31 01:15:14.150915772 +0000 UTC m=+1158.509638085" Oct 31 01:15:14.862291 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:14.862248 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:15:15.145113 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:15.145019 4440 generic.go:356] "Generic (PLEG): container finished" podID="877c89f5-20ff-4b40-a161-15b6e135c4b9" containerID="b326cc73957bde163c0d4d39d56898cbc1b701f80d5a611f3d6aa0fb4ad08aa4" exitCode=60 Oct 31 01:15:15.145113 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:15.145095 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-487a6c-29890155-gpd4t" event={"ID":"877c89f5-20ff-4b40-a161-15b6e135c4b9","Type":"ContainerDied","Data":"b326cc73957bde163c0d4d39d56898cbc1b701f80d5a611f3d6aa0fb4ad08aa4"} Oct 31 01:15:15.145580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:15.145162 4440 scope.go:122] "RemoveContainer" containerID="bf08e676cb9ef26c7c0c9f4f1b9c3a35f6a3f7d064e8cce5da6e499b57a13d4a" Oct 31 01:15:15.145621 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:15.145581 4440 scope.go:122] "RemoveContainer" containerID="b326cc73957bde163c0d4d39d56898cbc1b701f80d5a611f3d6aa0fb4ad08aa4" Oct 31 01:15:15.145912 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:15:15.145884 4440 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-487a6c-29890155-gpd4t_odh-ai-gateway-infra(877c89f5-20ff-4b40-a161-15b6e135c4b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-487a6c-29890155-gpd4t" podUID="877c89f5-20ff-4b40-a161-15b6e135c4b9" Oct 31 01:15:15.161017 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:15.160987 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-487a6c-29890155-gpd4t"] Oct 31 01:15:16.310806 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:16.310778 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-487a6c-29890155-gpd4t" Oct 31 01:15:16.370548 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:16.370504 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-487a6c-6f784564bc-c6qb5" podUID="0a3e8a39-48ca-4063-93b8-8afa2f1f18ac" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:15:16.431506 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:16.431471 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/877c89f5-20ff-4b40-a161-15b6e135c4b9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/877c89f5-20ff-4b40-a161-15b6e135c4b9-maas-api-service-ca\") pod \"877c89f5-20ff-4b40-a161-15b6e135c4b9\" (UID: \"877c89f5-20ff-4b40-a161-15b6e135c4b9\") " Oct 31 01:15:16.431722 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:16.431590 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/877c89f5-20ff-4b40-a161-15b6e135c4b9-kube-api-access-tjsmv\" (UniqueName: \"kubernetes.io/projected/877c89f5-20ff-4b40-a161-15b6e135c4b9-kube-api-access-tjsmv\") pod \"877c89f5-20ff-4b40-a161-15b6e135c4b9\" (UID: \"877c89f5-20ff-4b40-a161-15b6e135c4b9\") " Oct 31 01:15:16.431811 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:16.431791 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/877c89f5-20ff-4b40-a161-15b6e135c4b9-maas-api-service-ca" pod "877c89f5-20ff-4b40-a161-15b6e135c4b9" (UID: "877c89f5-20ff-4b40-a161-15b6e135c4b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:16.432024 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:16.432008 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/877c89f5-20ff-4b40-a161-15b6e135c4b9-maas-api-service-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:16.433732 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:16.433707 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/877c89f5-20ff-4b40-a161-15b6e135c4b9-kube-api-access-tjsmv" pod "877c89f5-20ff-4b40-a161-15b6e135c4b9" (UID: "877c89f5-20ff-4b40-a161-15b6e135c4b9"). InnerVolumeSpecName "kube-api-access-tjsmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:16.533098 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:16.533060 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjsmv\" (UniqueName: \"kubernetes.io/projected/877c89f5-20ff-4b40-a161-15b6e135c4b9-kube-api-access-tjsmv\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:16.764577 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:16.764479 4440 scope.go:122] "RemoveContainer" containerID="7e2dd1609cc0bb326e96183017f39a031064731ca3c9b553784ed47a3bb212cb" Oct 31 01:15:17.164051 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:17.164010 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-kdh9z" event={"ID":"c7c55c28-c3ca-4832-9559-b05e99a3995d","Type":"ContainerStarted","Data":"83ea23cec5893606324ac8a1f4b8c01a83901810900cb0b21cb2f9232f8f4ac2"} Oct 31 01:15:17.166335 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:17.166202 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-487a6c-29890155-gpd4t" event={"ID":"877c89f5-20ff-4b40-a161-15b6e135c4b9","Type":"ContainerDied","Data":"7c0ee9ae135a3430cd3268a08780c0c084703fc68762ed925210d16c6614cbaa"} Oct 31 01:15:17.166335 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:17.166266 4440 scope.go:122] "RemoveContainer" containerID="b326cc73957bde163c0d4d39d56898cbc1b701f80d5a611f3d6aa0fb4ad08aa4" Oct 31 01:15:17.166335 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:17.166266 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-487a6c-29890155-gpd4t" Oct 31 01:15:17.207193 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:17.207155 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-487a6c-29890155-gpd4t"] Oct 31 01:15:17.210265 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:17.210234 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-487a6c-29890155-gpd4t"] Oct 31 01:15:17.775788 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:17.775750 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="877c89f5-20ff-4b40-a161-15b6e135c4b9" path="/var/lib/kubelet/pods/877c89f5-20ff-4b40-a161-15b6e135c4b9/volumes" Oct 31 01:15:17.850831 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:17.850769 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:17.984522 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:17.982432 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:18.102053 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:18.102013 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:18.179957 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:18.179916 4440 generic.go:356] "Generic (PLEG): container finished" podID="c7c55c28-c3ca-4832-9559-b05e99a3995d" containerID="83ea23cec5893606324ac8a1f4b8c01a83901810900cb0b21cb2f9232f8f4ac2" exitCode=60 Oct 31 01:15:18.180282 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:18.180006 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-kdh9z" event={"ID":"c7c55c28-c3ca-4832-9559-b05e99a3995d","Type":"ContainerDied","Data":"83ea23cec5893606324ac8a1f4b8c01a83901810900cb0b21cb2f9232f8f4ac2"} Oct 31 01:15:18.180399 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:18.180370 4440 scope.go:122] "RemoveContainer" containerID="7e2dd1609cc0bb326e96183017f39a031064731ca3c9b553784ed47a3bb212cb" Oct 31 01:15:18.192508 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:18.181168 4440 scope.go:122] "RemoveContainer" containerID="83ea23cec5893606324ac8a1f4b8c01a83901810900cb0b21cb2f9232f8f4ac2" Oct 31 01:15:18.192508 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:15:18.181565 4440 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890155-kdh9z_odh-ai-gateway-infra(c7c55c28-c3ca-4832-9559-b05e99a3995d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-kdh9z" podUID="c7c55c28-c3ca-4832-9559-b05e99a3995d" Oct 31 01:15:18.216442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:18.206652 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890155-kdh9z"] Oct 31 01:15:18.250135 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:18.250030 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:18.482180 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:18.482120 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" containerID="cri-o://a011701ce2a965b13374b7fe0d5c19a20d52c7217b49af41afd83e75c8f875c4" gracePeriod=45 Oct 31 01:15:18.734542 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:18.734502 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:18.738582 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:18.738549 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" Oct 31 01:15:18.860041 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:18.859934 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/054f9e47-675f-4722-b743-79bbdc826585-tls-certs\" (UniqueName: \"kubernetes.io/secret/054f9e47-675f-4722-b743-79bbdc826585-tls-certs\") pod \"054f9e47-675f-4722-b743-79bbdc826585\" (UID: \"054f9e47-675f-4722-b743-79bbdc826585\") " Oct 31 01:15:18.860536 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:18.860086 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/054f9e47-675f-4722-b743-79bbdc826585-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/054f9e47-675f-4722-b743-79bbdc826585-model-cache\") pod \"054f9e47-675f-4722-b743-79bbdc826585\" (UID: \"054f9e47-675f-4722-b743-79bbdc826585\") " Oct 31 01:15:18.860536 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:18.860238 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/054f9e47-675f-4722-b743-79bbdc826585-kube-api-access-bz4kp\" (UniqueName: \"kubernetes.io/projected/054f9e47-675f-4722-b743-79bbdc826585-kube-api-access-bz4kp\") pod \"054f9e47-675f-4722-b743-79bbdc826585\" (UID: \"054f9e47-675f-4722-b743-79bbdc826585\") " Oct 31 01:15:18.860536 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:18.860307 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/054f9e47-675f-4722-b743-79bbdc826585-home\" (UniqueName: \"kubernetes.io/empty-dir/054f9e47-675f-4722-b743-79bbdc826585-home\") pod \"054f9e47-675f-4722-b743-79bbdc826585\" (UID: \"054f9e47-675f-4722-b743-79bbdc826585\") " Oct 31 01:15:18.860536 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:18.860387 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054f9e47-675f-4722-b743-79bbdc826585-model-cache" pod "054f9e47-675f-4722-b743-79bbdc826585" (UID: "054f9e47-675f-4722-b743-79bbdc826585"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:18.860536 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:18.860465 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/054f9e47-675f-4722-b743-79bbdc826585-dshm\" (UniqueName: \"kubernetes.io/empty-dir/054f9e47-675f-4722-b743-79bbdc826585-dshm\") pod \"054f9e47-675f-4722-b743-79bbdc826585\" (UID: \"054f9e47-675f-4722-b743-79bbdc826585\") " Oct 31 01:15:18.860536 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:18.860530 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/054f9e47-675f-4722-b743-79bbdc826585-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/054f9e47-675f-4722-b743-79bbdc826585-tmp-dir\") pod \"054f9e47-675f-4722-b743-79bbdc826585\" (UID: \"054f9e47-675f-4722-b743-79bbdc826585\") " Oct 31 01:15:18.860897 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:18.860572 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054f9e47-675f-4722-b743-79bbdc826585-home" pod "054f9e47-675f-4722-b743-79bbdc826585" (UID: "054f9e47-675f-4722-b743-79bbdc826585"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:18.860965 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:18.860947 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054f9e47-675f-4722-b743-79bbdc826585-tmp-dir" pod "054f9e47-675f-4722-b743-79bbdc826585" (UID: "054f9e47-675f-4722-b743-79bbdc826585"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:18.861072 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:18.861043 4440 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/054f9e47-675f-4722-b743-79bbdc826585-home\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:18.861165 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:18.861076 4440 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/054f9e47-675f-4722-b743-79bbdc826585-model-cache\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:18.862577 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:18.862531 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/054f9e47-675f-4722-b743-79bbdc826585-tls-certs" pod "054f9e47-675f-4722-b743-79bbdc826585" (UID: "054f9e47-675f-4722-b743-79bbdc826585"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:15:18.863041 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:18.863004 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054f9e47-675f-4722-b743-79bbdc826585-dshm" pod "054f9e47-675f-4722-b743-79bbdc826585" (UID: "054f9e47-675f-4722-b743-79bbdc826585"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:18.863390 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:18.863373 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/054f9e47-675f-4722-b743-79bbdc826585-kube-api-access-bz4kp" pod "054f9e47-675f-4722-b743-79bbdc826585" (UID: "054f9e47-675f-4722-b743-79bbdc826585"). InnerVolumeSpecName "kube-api-access-bz4kp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:18.961721 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:18.961681 4440 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/054f9e47-675f-4722-b743-79bbdc826585-tls-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:18.961721 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:18.961715 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz4kp\" (UniqueName: \"kubernetes.io/projected/054f9e47-675f-4722-b743-79bbdc826585-kube-api-access-bz4kp\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:18.961932 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:18.961734 4440 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/054f9e47-675f-4722-b743-79bbdc826585-dshm\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:18.961932 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:18.961749 4440 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/054f9e47-675f-4722-b743-79bbdc826585-tmp-dir\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:19.190815 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:19.190779 4440 generic.go:356] "Generic (PLEG): container finished" podID="054f9e47-675f-4722-b743-79bbdc826585" containerID="a011701ce2a965b13374b7fe0d5c19a20d52c7217b49af41afd83e75c8f875c4" exitCode=0 Oct 31 01:15:19.190815 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:19.190812 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" Oct 31 01:15:19.190969 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:19.190854 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" event={"ID":"054f9e47-675f-4722-b743-79bbdc826585","Type":"ContainerDied","Data":"a011701ce2a965b13374b7fe0d5c19a20d52c7217b49af41afd83e75c8f875c4"} Oct 31 01:15:19.190969 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:19.190904 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg" event={"ID":"054f9e47-675f-4722-b743-79bbdc826585","Type":"ContainerDied","Data":"f6e00286ab6fa244240117d3a981d1c5e36e193d853adbb39a7632b12bb35698"} Oct 31 01:15:19.190969 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:19.190923 4440 scope.go:122] "RemoveContainer" containerID="a011701ce2a965b13374b7fe0d5c19a20d52c7217b49af41afd83e75c8f875c4" Oct 31 01:15:19.202076 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:19.201843 4440 scope.go:122] "RemoveContainer" containerID="a011701ce2a965b13374b7fe0d5c19a20d52c7217b49af41afd83e75c8f875c4" Oct 31 01:15:19.202347 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:15:19.202160 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a011701ce2a965b13374b7fe0d5c19a20d52c7217b49af41afd83e75c8f875c4\": container with ID starting with a011701ce2a965b13374b7fe0d5c19a20d52c7217b49af41afd83e75c8f875c4 not found: ID does not exist" containerID="a011701ce2a965b13374b7fe0d5c19a20d52c7217b49af41afd83e75c8f875c4" Oct 31 01:15:19.202347 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:19.202199 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a011701ce2a965b13374b7fe0d5c19a20d52c7217b49af41afd83e75c8f875c4"} err="failed to get container status \"a011701ce2a965b13374b7fe0d5c19a20d52c7217b49af41afd83e75c8f875c4\": rpc error: code = NotFound desc = could not find container \"a011701ce2a965b13374b7fe0d5c19a20d52c7217b49af41afd83e75c8f875c4\": container with ID starting with a011701ce2a965b13374b7fe0d5c19a20d52c7217b49af41afd83e75c8f875c4 not found: ID does not exist" Oct 31 01:15:19.224004 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:19.223943 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg"] Oct 31 01:15:19.229110 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:19.229077 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-487a6c/distinct-w3-487a6c-kserve-6b56b54dd5-mqtxg"] Oct 31 01:15:19.366218 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:19.366191 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-kdh9z" Oct 31 01:15:19.468132 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:19.468041 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7c55c28-c3ca-4832-9559-b05e99a3995d-kube-api-access-s9nsg\" (UniqueName: \"kubernetes.io/projected/c7c55c28-c3ca-4832-9559-b05e99a3995d-kube-api-access-s9nsg\") pod \"c7c55c28-c3ca-4832-9559-b05e99a3995d\" (UID: \"c7c55c28-c3ca-4832-9559-b05e99a3995d\") " Oct 31 01:15:19.468132 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:19.468097 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7c55c28-c3ca-4832-9559-b05e99a3995d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7c55c28-c3ca-4832-9559-b05e99a3995d-maas-api-service-ca\") pod \"c7c55c28-c3ca-4832-9559-b05e99a3995d\" (UID: \"c7c55c28-c3ca-4832-9559-b05e99a3995d\") " Oct 31 01:15:19.468535 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:19.468506 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7c55c28-c3ca-4832-9559-b05e99a3995d-maas-api-service-ca" pod "c7c55c28-c3ca-4832-9559-b05e99a3995d" (UID: "c7c55c28-c3ca-4832-9559-b05e99a3995d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:19.468665 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:19.468647 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7c55c28-c3ca-4832-9559-b05e99a3995d-maas-api-service-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:19.470241 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:19.470210 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7c55c28-c3ca-4832-9559-b05e99a3995d-kube-api-access-s9nsg" pod "c7c55c28-c3ca-4832-9559-b05e99a3995d" (UID: "c7c55c28-c3ca-4832-9559-b05e99a3995d"). InnerVolumeSpecName "kube-api-access-s9nsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:19.569673 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:19.569623 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9nsg\" (UniqueName: \"kubernetes.io/projected/c7c55c28-c3ca-4832-9559-b05e99a3995d-kube-api-access-s9nsg\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:19.771301 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:19.771253 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="054f9e47-675f-4722-b743-79bbdc826585" path="/var/lib/kubelet/pods/054f9e47-675f-4722-b743-79bbdc826585/volumes" Oct 31 01:15:20.201401 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:20.201368 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-kdh9z" Oct 31 01:15:20.201946 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:20.201507 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-kdh9z" event={"ID":"c7c55c28-c3ca-4832-9559-b05e99a3995d","Type":"ContainerDied","Data":"4a0a83afd9685fed4b8892cc23314549b85cdb0274712b361bd95994e7ad61d0"} Oct 31 01:15:20.201946 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:20.201577 4440 scope.go:122] "RemoveContainer" containerID="83ea23cec5893606324ac8a1f4b8c01a83901810900cb0b21cb2f9232f8f4ac2" Oct 31 01:15:20.224353 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:20.224309 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890155-kdh9z"] Oct 31 01:15:20.231725 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:20.231688 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890155-kdh9z"] Oct 31 01:15:21.773477 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:21.773408 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7c55c28-c3ca-4832-9559-b05e99a3995d" path="/var/lib/kubelet/pods/c7c55c28-c3ca-4832-9559-b05e99a3995d/volumes" Oct 31 01:15:23.256751 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:23.256718 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:15:24.362445 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:24.359622 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:24.775467 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:24.771639 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:25.300997 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:25.300963 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:25.406797 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:25.406764 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:25.874229 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:25.874189 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:26.371349 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:26.371307 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-487a6c-6f784564bc-c6qb5" podUID="0a3e8a39-48ca-4063-93b8-8afa2f1f18ac" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:15:26.918879 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:26.918840 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:27.027068 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:27.027028 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:30.755207 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:30.755158 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:15:31.003684 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:15:31.003640 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:15:31.862356 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:31.862316 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:31.985455 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:31.985411 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:32.145539 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:32.145441 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz"] Oct 31 01:15:32.145730 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:32.145700 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" podUID="f8555032-1556-4f20-a9bb-7f1dcbf99dcc" containerName="istio-proxy" containerID="cri-o://829a6af953a02de438e6a74379be137cd8a51cadc3ec4a685b782715a031d660" gracePeriod=30 Oct 31 01:15:32.163663 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:32.163632 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz"] Oct 31 01:15:32.164103 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:32.164076 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" podUID="a2ef7da0-2a19-4794-a80f-9badb9461e9a" containerName="istio-proxy" containerID="cri-o://29979f9137e8c116558b7b16cfac14685809a3092cad8481902db8d834367e07" gracePeriod=30 Oct 31 01:15:32.173237 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:32.173169 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br"] Oct 31 01:15:32.173559 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:32.173396 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" podUID="66b984db-1113-44af-b6d9-6cf452c97187" containerName="istio-proxy" containerID="cri-o://8283803af9dca21d4b672094bf290e6bab2c020f091996b1a51930df9bdb16aa" gracePeriod=30 Oct 31 01:15:32.180527 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:32.180504 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw"] Oct 31 01:15:32.180699 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:32.180682 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" podUID="613c5424-a9e2-43da-95aa-807e2b99ab2e" containerName="istio-proxy" containerID="cri-o://babe783bf64843c77c73c95295c0ff7e57a2de42a16a3f1e9d96e5e2820cd290" gracePeriod=30 Oct 31 01:15:32.227556 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:32.227504 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:34.063872 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:34.063798 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-487a6c-6f784564bc-c6qb5" Oct 31 01:15:34.160316 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:34.160268 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a3e8a39-48ca-4063-93b8-8afa2f1f18ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a3e8a39-48ca-4063-93b8-8afa2f1f18ac-plugins-config-volume\") pod \"0a3e8a39-48ca-4063-93b8-8afa2f1f18ac\" (UID: \"0a3e8a39-48ca-4063-93b8-8afa2f1f18ac\") " Oct 31 01:15:34.160523 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:34.160377 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a3e8a39-48ca-4063-93b8-8afa2f1f18ac-kube-api-access-qcvrc\" (UniqueName: \"kubernetes.io/projected/0a3e8a39-48ca-4063-93b8-8afa2f1f18ac-kube-api-access-qcvrc\") pod \"0a3e8a39-48ca-4063-93b8-8afa2f1f18ac\" (UID: \"0a3e8a39-48ca-4063-93b8-8afa2f1f18ac\") " Oct 31 01:15:34.160770 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:34.160742 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a3e8a39-48ca-4063-93b8-8afa2f1f18ac-plugins-config-volume" pod "0a3e8a39-48ca-4063-93b8-8afa2f1f18ac" (UID: "0a3e8a39-48ca-4063-93b8-8afa2f1f18ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:34.162702 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:34.162670 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a3e8a39-48ca-4063-93b8-8afa2f1f18ac-kube-api-access-qcvrc" pod "0a3e8a39-48ca-4063-93b8-8afa2f1f18ac" (UID: "0a3e8a39-48ca-4063-93b8-8afa2f1f18ac"). InnerVolumeSpecName "kube-api-access-qcvrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:34.261933 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:34.261889 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcvrc\" (UniqueName: \"kubernetes.io/projected/0a3e8a39-48ca-4063-93b8-8afa2f1f18ac-kube-api-access-qcvrc\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:34.261933 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:34.261929 4440 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a3e8a39-48ca-4063-93b8-8afa2f1f18ac-plugins-config-volume\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:34.365199 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:34.365111 4440 generic.go:356] "Generic (PLEG): container finished" podID="0a3e8a39-48ca-4063-93b8-8afa2f1f18ac" containerID="03b0b7a38b56730c846fed5a5f497ad2d1a8b593d453e4e09eb61e8e5182d098" exitCode=0 Oct 31 01:15:34.365199 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:34.365156 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-487a6c-6f784564bc-c6qb5" Oct 31 01:15:34.365389 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:34.365192 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-487a6c-6f784564bc-c6qb5" event={"ID":"0a3e8a39-48ca-4063-93b8-8afa2f1f18ac","Type":"ContainerDied","Data":"03b0b7a38b56730c846fed5a5f497ad2d1a8b593d453e4e09eb61e8e5182d098"} Oct 31 01:15:34.365389 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:34.365251 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-487a6c-6f784564bc-c6qb5" event={"ID":"0a3e8a39-48ca-4063-93b8-8afa2f1f18ac","Type":"ContainerDied","Data":"74a76aa300f01e1b738e0fc3c46a6dce5cdf1c9562d4cfc6ccd433c36e097cbf"} Oct 31 01:15:34.365389 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:34.365275 4440 scope.go:122] "RemoveContainer" containerID="03b0b7a38b56730c846fed5a5f497ad2d1a8b593d453e4e09eb61e8e5182d098" Oct 31 01:15:34.375568 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:34.375541 4440 scope.go:122] "RemoveContainer" containerID="03b0b7a38b56730c846fed5a5f497ad2d1a8b593d453e4e09eb61e8e5182d098" Oct 31 01:15:34.375894 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:15:34.375875 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03b0b7a38b56730c846fed5a5f497ad2d1a8b593d453e4e09eb61e8e5182d098\": container with ID starting with 03b0b7a38b56730c846fed5a5f497ad2d1a8b593d453e4e09eb61e8e5182d098 not found: ID does not exist" containerID="03b0b7a38b56730c846fed5a5f497ad2d1a8b593d453e4e09eb61e8e5182d098" Oct 31 01:15:34.375942 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:34.375905 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03b0b7a38b56730c846fed5a5f497ad2d1a8b593d453e4e09eb61e8e5182d098"} err="failed to get container status \"03b0b7a38b56730c846fed5a5f497ad2d1a8b593d453e4e09eb61e8e5182d098\": rpc error: code = NotFound desc = could not find container \"03b0b7a38b56730c846fed5a5f497ad2d1a8b593d453e4e09eb61e8e5182d098\": container with ID starting with 03b0b7a38b56730c846fed5a5f497ad2d1a8b593d453e4e09eb61e8e5182d098 not found: ID does not exist" Oct 31 01:15:34.403238 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:34.403204 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-487a6c-6f784564bc-c6qb5"] Oct 31 01:15:34.411839 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:34.411806 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-487a6c-6f784564bc-c6qb5"] Oct 31 01:15:34.426928 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:34.426893 4440 patch_prober.go:29] interesting pod/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:15:34.427104 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:34.426951 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" podUID="613c5424-a9e2-43da-95aa-807e2b99ab2e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:15:35.155499 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:35.155457 4440 patch_prober.go:29] interesting pod/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:15:35.155996 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:35.155525 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" podUID="a2ef7da0-2a19-4794-a80f-9badb9461e9a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:15:35.504324 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:35.504286 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:15:35.773011 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:35.772922 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a3e8a39-48ca-4063-93b8-8afa2f1f18ac" path="/var/lib/kubelet/pods/0a3e8a39-48ca-4063-93b8-8afa2f1f18ac/volumes" Oct 31 01:15:37.300557 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:37.282143 4440 patch_prober.go:29] interesting pod/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 31 01:15:37.300557 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:37.282220 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" podUID="66b984db-1113-44af-b6d9-6cf452c97187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 31 01:15:37.488694 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:37.488137 4440 generic.go:356] "Generic (PLEG): container finished" podID="613c5424-a9e2-43da-95aa-807e2b99ab2e" containerID="babe783bf64843c77c73c95295c0ff7e57a2de42a16a3f1e9d96e5e2820cd290" exitCode=0 Oct 31 01:15:37.488694 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:37.488227 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" event={"ID":"613c5424-a9e2-43da-95aa-807e2b99ab2e","Type":"ContainerDied","Data":"babe783bf64843c77c73c95295c0ff7e57a2de42a16a3f1e9d96e5e2820cd290"} Oct 31 01:15:37.509795 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:37.509754 4440 generic.go:356] "Generic (PLEG): container finished" podID="f8555032-1556-4f20-a9bb-7f1dcbf99dcc" containerID="829a6af953a02de438e6a74379be137cd8a51cadc3ec4a685b782715a031d660" exitCode=0 Oct 31 01:15:37.513577 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:37.512469 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" event={"ID":"f8555032-1556-4f20-a9bb-7f1dcbf99dcc","Type":"ContainerDied","Data":"829a6af953a02de438e6a74379be137cd8a51cadc3ec4a685b782715a031d660"} Oct 31 01:15:37.513869 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:37.513847 4440 scope.go:122] "RemoveContainer" containerID="d224ea1ef076b80805213e38b80de1bc55b5b9f72bf1d142e023460bd3573b22" Oct 31 01:15:38.728998 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:38.728958 4440 generic.go:356] "Generic (PLEG): container finished" podID="66b984db-1113-44af-b6d9-6cf452c97187" containerID="8283803af9dca21d4b672094bf290e6bab2c020f091996b1a51930df9bdb16aa" exitCode=0 Oct 31 01:15:38.729730 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:38.729696 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" event={"ID":"66b984db-1113-44af-b6d9-6cf452c97187","Type":"ContainerDied","Data":"8283803af9dca21d4b672094bf290e6bab2c020f091996b1a51930df9bdb16aa"} Oct 31 01:15:38.909617 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:38.827163 4440 generic.go:356] "Generic (PLEG): container finished" podID="a2ef7da0-2a19-4794-a80f-9badb9461e9a" containerID="29979f9137e8c116558b7b16cfac14685809a3092cad8481902db8d834367e07" exitCode=0 Oct 31 01:15:38.909617 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:38.827325 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" event={"ID":"a2ef7da0-2a19-4794-a80f-9badb9461e9a","Type":"ContainerDied","Data":"29979f9137e8c116558b7b16cfac14685809a3092cad8481902db8d834367e07"} Oct 31 01:15:39.220440 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.219815 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" Oct 31 01:15:39.267442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.248848 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-kube-api-access-tvx8c\" (UniqueName: \"kubernetes.io/projected/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-kube-api-access-tvx8c\") pod \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\" (UID: \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\") " Oct 31 01:15:39.267442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.248961 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-workload-certs\") pod \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\" (UID: \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\") " Oct 31 01:15:39.267442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.249014 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-istio-data\") pod \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\" (UID: \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\") " Oct 31 01:15:39.267442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.249069 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-credential-socket\") pod \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\" (UID: \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\") " Oct 31 01:15:39.267442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.249182 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-istio-token\" (UniqueName: \"kubernetes.io/projected/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-istio-token\") pod \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\" (UID: \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\") " Oct 31 01:15:39.267442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.249236 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-workload-socket\") pod \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\" (UID: \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\") " Oct 31 01:15:39.267442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.249290 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-istio-envoy\") pod \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\" (UID: \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\") " Oct 31 01:15:39.267442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.249325 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-istiod-ca-cert\") pod \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\" (UID: \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\") " Oct 31 01:15:39.267442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.249372 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-istio-podinfo\") pod \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\" (UID: \"f8555032-1556-4f20-a9bb-7f1dcbf99dcc\") " Oct 31 01:15:39.267442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.251571 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-workload-certs" pod "f8555032-1556-4f20-a9bb-7f1dcbf99dcc" (UID: "f8555032-1556-4f20-a9bb-7f1dcbf99dcc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:39.267442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.252360 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-credential-socket" pod "f8555032-1556-4f20-a9bb-7f1dcbf99dcc" (UID: "f8555032-1556-4f20-a9bb-7f1dcbf99dcc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:39.267442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.252692 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-workload-socket" pod "f8555032-1556-4f20-a9bb-7f1dcbf99dcc" (UID: "f8555032-1556-4f20-a9bb-7f1dcbf99dcc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:39.267442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.253650 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-istio-data" pod "f8555032-1556-4f20-a9bb-7f1dcbf99dcc" (UID: "f8555032-1556-4f20-a9bb-7f1dcbf99dcc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:39.267442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.257799 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-istiod-ca-cert" pod "f8555032-1556-4f20-a9bb-7f1dcbf99dcc" (UID: "f8555032-1556-4f20-a9bb-7f1dcbf99dcc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:39.267442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.257955 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" Oct 31 01:15:39.267442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.260554 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" Oct 31 01:15:39.274457 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.270291 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" Oct 31 01:15:39.275134 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.274932 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-istio-token" pod "f8555032-1556-4f20-a9bb-7f1dcbf99dcc" (UID: "f8555032-1556-4f20-a9bb-7f1dcbf99dcc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:39.275134 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.275091 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-istio-podinfo" pod "f8555032-1556-4f20-a9bb-7f1dcbf99dcc" (UID: "f8555032-1556-4f20-a9bb-7f1dcbf99dcc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:15:39.380603 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.375604 4440 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-istio-token\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:39.380603 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.375646 4440 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-workload-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:39.380603 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.375663 4440 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-istiod-ca-cert\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:39.380603 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.375678 4440 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-istio-podinfo\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:39.380603 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.375700 4440 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-workload-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:39.380603 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.375716 4440 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-istio-data\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:39.380603 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.375730 4440 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-credential-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:39.436498 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.436360 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-kube-api-access-tvx8c" pod "f8555032-1556-4f20-a9bb-7f1dcbf99dcc" (UID: "f8555032-1556-4f20-a9bb-7f1dcbf99dcc"). InnerVolumeSpecName "kube-api-access-tvx8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:39.446147 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.446105 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-istio-envoy" pod "f8555032-1556-4f20-a9bb-7f1dcbf99dcc" (UID: "f8555032-1556-4f20-a9bb-7f1dcbf99dcc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:39.476148 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.476096 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66b984db-1113-44af-b6d9-6cf452c97187-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66b984db-1113-44af-b6d9-6cf452c97187-istio-envoy\") pod \"66b984db-1113-44af-b6d9-6cf452c97187\" (UID: \"66b984db-1113-44af-b6d9-6cf452c97187\") " Oct 31 01:15:39.476374 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.476156 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2ef7da0-2a19-4794-a80f-9badb9461e9a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2ef7da0-2a19-4794-a80f-9badb9461e9a-istiod-ca-cert\") pod \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\" (UID: \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\") " Oct 31 01:15:39.476374 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.476188 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/613c5424-a9e2-43da-95aa-807e2b99ab2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/613c5424-a9e2-43da-95aa-807e2b99ab2e-workload-socket\") pod \"613c5424-a9e2-43da-95aa-807e2b99ab2e\" (UID: \"613c5424-a9e2-43da-95aa-807e2b99ab2e\") " Oct 31 01:15:39.476374 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.476221 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2ef7da0-2a19-4794-a80f-9badb9461e9a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ef7da0-2a19-4794-a80f-9badb9461e9a-workload-socket\") pod \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\" (UID: \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\") " Oct 31 01:15:39.476374 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.476249 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2ef7da0-2a19-4794-a80f-9badb9461e9a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ef7da0-2a19-4794-a80f-9badb9461e9a-credential-socket\") pod \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\" (UID: \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\") " Oct 31 01:15:39.476374 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.476296 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66b984db-1113-44af-b6d9-6cf452c97187-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b984db-1113-44af-b6d9-6cf452c97187-workload-socket\") pod \"66b984db-1113-44af-b6d9-6cf452c97187\" (UID: \"66b984db-1113-44af-b6d9-6cf452c97187\") " Oct 31 01:15:39.476374 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.476330 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66b984db-1113-44af-b6d9-6cf452c97187-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66b984db-1113-44af-b6d9-6cf452c97187-workload-certs\") pod \"66b984db-1113-44af-b6d9-6cf452c97187\" (UID: \"66b984db-1113-44af-b6d9-6cf452c97187\") " Oct 31 01:15:39.476374 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.476362 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/613c5424-a9e2-43da-95aa-807e2b99ab2e-kube-api-access-8hq9d\" (UniqueName: \"kubernetes.io/projected/613c5424-a9e2-43da-95aa-807e2b99ab2e-kube-api-access-8hq9d\") pod \"613c5424-a9e2-43da-95aa-807e2b99ab2e\" (UID: \"613c5424-a9e2-43da-95aa-807e2b99ab2e\") " Oct 31 01:15:39.476787 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.476393 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2ef7da0-2a19-4794-a80f-9badb9461e9a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2ef7da0-2a19-4794-a80f-9badb9461e9a-istio-envoy\") pod \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\" (UID: \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\") " Oct 31 01:15:39.476787 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.476448 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2ef7da0-2a19-4794-a80f-9badb9461e9a-kube-api-access-zmk9b\" (UniqueName: \"kubernetes.io/projected/a2ef7da0-2a19-4794-a80f-9badb9461e9a-kube-api-access-zmk9b\") pod \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\" (UID: \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\") " Oct 31 01:15:39.476787 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.476498 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/613c5424-a9e2-43da-95aa-807e2b99ab2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/613c5424-a9e2-43da-95aa-807e2b99ab2e-credential-socket\") pod \"613c5424-a9e2-43da-95aa-807e2b99ab2e\" (UID: \"613c5424-a9e2-43da-95aa-807e2b99ab2e\") " Oct 31 01:15:39.476787 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.476533 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66b984db-1113-44af-b6d9-6cf452c97187-kube-api-access-psdp9\" (UniqueName: \"kubernetes.io/projected/66b984db-1113-44af-b6d9-6cf452c97187-kube-api-access-psdp9\") pod \"66b984db-1113-44af-b6d9-6cf452c97187\" (UID: \"66b984db-1113-44af-b6d9-6cf452c97187\") " Oct 31 01:15:39.476787 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.476575 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66b984db-1113-44af-b6d9-6cf452c97187-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66b984db-1113-44af-b6d9-6cf452c97187-istiod-ca-cert\") pod \"66b984db-1113-44af-b6d9-6cf452c97187\" (UID: \"66b984db-1113-44af-b6d9-6cf452c97187\") " Oct 31 01:15:39.476787 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.476608 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/613c5424-a9e2-43da-95aa-807e2b99ab2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/613c5424-a9e2-43da-95aa-807e2b99ab2e-istiod-ca-cert\") pod \"613c5424-a9e2-43da-95aa-807e2b99ab2e\" (UID: \"613c5424-a9e2-43da-95aa-807e2b99ab2e\") " Oct 31 01:15:39.476787 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.476649 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/613c5424-a9e2-43da-95aa-807e2b99ab2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/613c5424-a9e2-43da-95aa-807e2b99ab2e-istio-podinfo\") pod \"613c5424-a9e2-43da-95aa-807e2b99ab2e\" (UID: \"613c5424-a9e2-43da-95aa-807e2b99ab2e\") " Oct 31 01:15:39.476787 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.476678 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66b984db-1113-44af-b6d9-6cf452c97187-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b984db-1113-44af-b6d9-6cf452c97187-credential-socket\") pod \"66b984db-1113-44af-b6d9-6cf452c97187\" (UID: \"66b984db-1113-44af-b6d9-6cf452c97187\") " Oct 31 01:15:39.476787 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.476740 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66b984db-1113-44af-b6d9-6cf452c97187-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66b984db-1113-44af-b6d9-6cf452c97187-istio-data\") pod \"66b984db-1113-44af-b6d9-6cf452c97187\" (UID: \"66b984db-1113-44af-b6d9-6cf452c97187\") " Oct 31 01:15:39.476787 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.476769 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2ef7da0-2a19-4794-a80f-9badb9461e9a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2ef7da0-2a19-4794-a80f-9badb9461e9a-workload-certs\") pod \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\" (UID: \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\") " Oct 31 01:15:39.477334 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.476801 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/613c5424-a9e2-43da-95aa-807e2b99ab2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/613c5424-a9e2-43da-95aa-807e2b99ab2e-istio-data\") pod \"613c5424-a9e2-43da-95aa-807e2b99ab2e\" (UID: \"613c5424-a9e2-43da-95aa-807e2b99ab2e\") " Oct 31 01:15:39.477334 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.476830 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2ef7da0-2a19-4794-a80f-9badb9461e9a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2ef7da0-2a19-4794-a80f-9badb9461e9a-istio-data\") pod \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\" (UID: \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\") " Oct 31 01:15:39.477334 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.476864 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2ef7da0-2a19-4794-a80f-9badb9461e9a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2ef7da0-2a19-4794-a80f-9badb9461e9a-istio-podinfo\") pod \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\" (UID: \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\") " Oct 31 01:15:39.477334 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.476943 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66b984db-1113-44af-b6d9-6cf452c97187-istio-token\" (UniqueName: \"kubernetes.io/projected/66b984db-1113-44af-b6d9-6cf452c97187-istio-token\") pod \"66b984db-1113-44af-b6d9-6cf452c97187\" (UID: \"66b984db-1113-44af-b6d9-6cf452c97187\") " Oct 31 01:15:39.477334 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.476982 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/613c5424-a9e2-43da-95aa-807e2b99ab2e-istio-token\" (UniqueName: \"kubernetes.io/projected/613c5424-a9e2-43da-95aa-807e2b99ab2e-istio-token\") pod \"613c5424-a9e2-43da-95aa-807e2b99ab2e\" (UID: \"613c5424-a9e2-43da-95aa-807e2b99ab2e\") " Oct 31 01:15:39.477334 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.477015 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66b984db-1113-44af-b6d9-6cf452c97187-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66b984db-1113-44af-b6d9-6cf452c97187-istio-podinfo\") pod \"66b984db-1113-44af-b6d9-6cf452c97187\" (UID: \"66b984db-1113-44af-b6d9-6cf452c97187\") " Oct 31 01:15:39.477334 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.477101 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/613c5424-a9e2-43da-95aa-807e2b99ab2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/613c5424-a9e2-43da-95aa-807e2b99ab2e-workload-certs\") pod \"613c5424-a9e2-43da-95aa-807e2b99ab2e\" (UID: \"613c5424-a9e2-43da-95aa-807e2b99ab2e\") " Oct 31 01:15:39.477334 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.477128 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2ef7da0-2a19-4794-a80f-9badb9461e9a-istio-token\" (UniqueName: \"kubernetes.io/projected/a2ef7da0-2a19-4794-a80f-9badb9461e9a-istio-token\") pod \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\" (UID: \"a2ef7da0-2a19-4794-a80f-9badb9461e9a\") " Oct 31 01:15:39.477334 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.477164 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/613c5424-a9e2-43da-95aa-807e2b99ab2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/613c5424-a9e2-43da-95aa-807e2b99ab2e-istio-envoy\") pod \"613c5424-a9e2-43da-95aa-807e2b99ab2e\" (UID: \"613c5424-a9e2-43da-95aa-807e2b99ab2e\") " Oct 31 01:15:39.477885 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.477575 4440 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-istio-envoy\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:39.477885 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.477594 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvx8c\" (UniqueName: \"kubernetes.io/projected/f8555032-1556-4f20-a9bb-7f1dcbf99dcc-kube-api-access-tvx8c\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:39.482510 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.482463 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2ef7da0-2a19-4794-a80f-9badb9461e9a-istio-data" pod "a2ef7da0-2a19-4794-a80f-9badb9461e9a" (UID: "a2ef7da0-2a19-4794-a80f-9badb9461e9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:39.482702 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.482586 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/613c5424-a9e2-43da-95aa-807e2b99ab2e-workload-certs" pod "613c5424-a9e2-43da-95aa-807e2b99ab2e" (UID: "613c5424-a9e2-43da-95aa-807e2b99ab2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:39.482865 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.482838 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/613c5424-a9e2-43da-95aa-807e2b99ab2e-istiod-ca-cert" pod "613c5424-a9e2-43da-95aa-807e2b99ab2e" (UID: "613c5424-a9e2-43da-95aa-807e2b99ab2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:39.483885 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.483857 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66b984db-1113-44af-b6d9-6cf452c97187-workload-socket" pod "66b984db-1113-44af-b6d9-6cf452c97187" (UID: "66b984db-1113-44af-b6d9-6cf452c97187"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:39.484815 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.484792 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66b984db-1113-44af-b6d9-6cf452c97187-istiod-ca-cert" pod "66b984db-1113-44af-b6d9-6cf452c97187" (UID: "66b984db-1113-44af-b6d9-6cf452c97187"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:39.486610 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.486552 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66b984db-1113-44af-b6d9-6cf452c97187-istio-envoy" pod "66b984db-1113-44af-b6d9-6cf452c97187" (UID: "66b984db-1113-44af-b6d9-6cf452c97187"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:39.486791 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.486745 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66b984db-1113-44af-b6d9-6cf452c97187-credential-socket" pod "66b984db-1113-44af-b6d9-6cf452c97187" (UID: "66b984db-1113-44af-b6d9-6cf452c97187"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:39.486953 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.486928 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66b984db-1113-44af-b6d9-6cf452c97187-istio-data" pod "66b984db-1113-44af-b6d9-6cf452c97187" (UID: "66b984db-1113-44af-b6d9-6cf452c97187"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:39.487252 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.487226 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/613c5424-a9e2-43da-95aa-807e2b99ab2e-istio-token" pod "613c5424-a9e2-43da-95aa-807e2b99ab2e" (UID: "613c5424-a9e2-43da-95aa-807e2b99ab2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:39.487504 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.487480 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ef7da0-2a19-4794-a80f-9badb9461e9a-istiod-ca-cert" pod "a2ef7da0-2a19-4794-a80f-9badb9461e9a" (UID: "a2ef7da0-2a19-4794-a80f-9badb9461e9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:39.487784 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.487628 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ef7da0-2a19-4794-a80f-9badb9461e9a-istio-token" pod "a2ef7da0-2a19-4794-a80f-9badb9461e9a" (UID: "a2ef7da0-2a19-4794-a80f-9badb9461e9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:39.487784 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.487647 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66b984db-1113-44af-b6d9-6cf452c97187-istio-podinfo" pod "66b984db-1113-44af-b6d9-6cf452c97187" (UID: "66b984db-1113-44af-b6d9-6cf452c97187"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:15:39.487939 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.487908 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/613c5424-a9e2-43da-95aa-807e2b99ab2e-workload-socket" pod "613c5424-a9e2-43da-95aa-807e2b99ab2e" (UID: "613c5424-a9e2-43da-95aa-807e2b99ab2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:39.488529 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.488043 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2ef7da0-2a19-4794-a80f-9badb9461e9a-workload-certs" pod "a2ef7da0-2a19-4794-a80f-9badb9461e9a" (UID: "a2ef7da0-2a19-4794-a80f-9badb9461e9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:39.488529 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.488119 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2ef7da0-2a19-4794-a80f-9badb9461e9a-workload-socket" pod "a2ef7da0-2a19-4794-a80f-9badb9461e9a" (UID: "a2ef7da0-2a19-4794-a80f-9badb9461e9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:39.488529 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.488251 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/613c5424-a9e2-43da-95aa-807e2b99ab2e-istio-data" pod "613c5424-a9e2-43da-95aa-807e2b99ab2e" (UID: "613c5424-a9e2-43da-95aa-807e2b99ab2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:39.488529 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.488312 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/613c5424-a9e2-43da-95aa-807e2b99ab2e-istio-podinfo" pod "613c5424-a9e2-43da-95aa-807e2b99ab2e" (UID: "613c5424-a9e2-43da-95aa-807e2b99ab2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:15:39.488828 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.488700 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/613c5424-a9e2-43da-95aa-807e2b99ab2e-credential-socket" pod "613c5424-a9e2-43da-95aa-807e2b99ab2e" (UID: "613c5424-a9e2-43da-95aa-807e2b99ab2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:39.489015 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.488993 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66b984db-1113-44af-b6d9-6cf452c97187-workload-certs" pod "66b984db-1113-44af-b6d9-6cf452c97187" (UID: "66b984db-1113-44af-b6d9-6cf452c97187"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:39.489231 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.489100 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2ef7da0-2a19-4794-a80f-9badb9461e9a-credential-socket" pod "a2ef7da0-2a19-4794-a80f-9badb9461e9a" (UID: "a2ef7da0-2a19-4794-a80f-9badb9461e9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:39.489231 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.489179 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2ef7da0-2a19-4794-a80f-9badb9461e9a-istio-podinfo" pod "a2ef7da0-2a19-4794-a80f-9badb9461e9a" (UID: "a2ef7da0-2a19-4794-a80f-9badb9461e9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:15:39.491082 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.490999 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/613c5424-a9e2-43da-95aa-807e2b99ab2e-kube-api-access-8hq9d" pod "613c5424-a9e2-43da-95aa-807e2b99ab2e" (UID: "613c5424-a9e2-43da-95aa-807e2b99ab2e"). InnerVolumeSpecName "kube-api-access-8hq9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:39.491488 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.491281 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66b984db-1113-44af-b6d9-6cf452c97187-istio-token" pod "66b984db-1113-44af-b6d9-6cf452c97187" (UID: "66b984db-1113-44af-b6d9-6cf452c97187"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:39.491488 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.491410 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/613c5424-a9e2-43da-95aa-807e2b99ab2e-istio-envoy" pod "613c5424-a9e2-43da-95aa-807e2b99ab2e" (UID: "613c5424-a9e2-43da-95aa-807e2b99ab2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:39.491692 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.491673 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ef7da0-2a19-4794-a80f-9badb9461e9a-kube-api-access-zmk9b" pod "a2ef7da0-2a19-4794-a80f-9badb9461e9a" (UID: "a2ef7da0-2a19-4794-a80f-9badb9461e9a"). InnerVolumeSpecName "kube-api-access-zmk9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:39.492595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.492567 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2ef7da0-2a19-4794-a80f-9badb9461e9a-istio-envoy" pod "a2ef7da0-2a19-4794-a80f-9badb9461e9a" (UID: "a2ef7da0-2a19-4794-a80f-9badb9461e9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:39.492990 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.492957 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66b984db-1113-44af-b6d9-6cf452c97187-kube-api-access-psdp9" pod "66b984db-1113-44af-b6d9-6cf452c97187" (UID: "66b984db-1113-44af-b6d9-6cf452c97187"). InnerVolumeSpecName "kube-api-access-psdp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:39.579404 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.579325 4440 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/613c5424-a9e2-43da-95aa-807e2b99ab2e-workload-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:39.579404 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.579367 4440 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2ef7da0-2a19-4794-a80f-9badb9461e9a-istio-token\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:39.579404 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.579379 4440 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/613c5424-a9e2-43da-95aa-807e2b99ab2e-istio-envoy\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:39.579404 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.579393 4440 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66b984db-1113-44af-b6d9-6cf452c97187-istio-envoy\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:39.579404 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.579407 4440 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2ef7da0-2a19-4794-a80f-9badb9461e9a-istiod-ca-cert\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:39.579404 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.579434 4440 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/613c5424-a9e2-43da-95aa-807e2b99ab2e-workload-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:39.579919 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.579448 4440 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ef7da0-2a19-4794-a80f-9badb9461e9a-workload-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:39.579919 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.579463 4440 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ef7da0-2a19-4794-a80f-9badb9461e9a-credential-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:39.579919 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.579478 4440 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b984db-1113-44af-b6d9-6cf452c97187-workload-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:39.579919 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.579490 4440 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66b984db-1113-44af-b6d9-6cf452c97187-workload-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:39.579919 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.579504 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hq9d\" (UniqueName: \"kubernetes.io/projected/613c5424-a9e2-43da-95aa-807e2b99ab2e-kube-api-access-8hq9d\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:39.579919 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.579519 4440 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2ef7da0-2a19-4794-a80f-9badb9461e9a-istio-envoy\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:39.579919 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.579532 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmk9b\" (UniqueName: \"kubernetes.io/projected/a2ef7da0-2a19-4794-a80f-9badb9461e9a-kube-api-access-zmk9b\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:39.579919 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.579544 4440 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/613c5424-a9e2-43da-95aa-807e2b99ab2e-credential-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:39.579919 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.579556 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psdp9\" (UniqueName: \"kubernetes.io/projected/66b984db-1113-44af-b6d9-6cf452c97187-kube-api-access-psdp9\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:39.579919 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.579569 4440 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66b984db-1113-44af-b6d9-6cf452c97187-istiod-ca-cert\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:39.579919 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.579581 4440 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/613c5424-a9e2-43da-95aa-807e2b99ab2e-istiod-ca-cert\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:39.579919 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.579594 4440 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/613c5424-a9e2-43da-95aa-807e2b99ab2e-istio-podinfo\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:39.579919 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.579606 4440 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b984db-1113-44af-b6d9-6cf452c97187-credential-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:39.579919 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.579620 4440 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66b984db-1113-44af-b6d9-6cf452c97187-istio-data\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:39.579919 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.579633 4440 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2ef7da0-2a19-4794-a80f-9badb9461e9a-workload-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:39.579919 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.579646 4440 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/613c5424-a9e2-43da-95aa-807e2b99ab2e-istio-data\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:39.579919 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.579662 4440 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2ef7da0-2a19-4794-a80f-9badb9461e9a-istio-data\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:39.579919 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.579674 4440 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2ef7da0-2a19-4794-a80f-9badb9461e9a-istio-podinfo\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:39.579919 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.579687 4440 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66b984db-1113-44af-b6d9-6cf452c97187-istio-token\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:39.579919 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.579702 4440 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/613c5424-a9e2-43da-95aa-807e2b99ab2e-istio-token\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:39.579919 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.579716 4440 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66b984db-1113-44af-b6d9-6cf452c97187-istio-podinfo\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:39.841758 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.841011 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" Oct 31 01:15:39.841758 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.841502 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw" event={"ID":"613c5424-a9e2-43da-95aa-807e2b99ab2e","Type":"ContainerDied","Data":"1167ebec6c98fdfd50951f4bbd10f8d8cad1cbcca7b18331dad032d916bd4bba"} Oct 31 01:15:39.841758 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.841565 4440 scope.go:122] "RemoveContainer" containerID="babe783bf64843c77c73c95295c0ff7e57a2de42a16a3f1e9d96e5e2820cd290" Oct 31 01:15:39.845288 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.845219 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:39.846298 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.846273 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" Oct 31 01:15:39.846858 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.846351 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz" event={"ID":"f8555032-1556-4f20-a9bb-7f1dcbf99dcc","Type":"ContainerDied","Data":"a619c6589a03293e2dcd26f57f0707a183d1b8bf009a7199854cab9252a502b4"} Oct 31 01:15:39.849388 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.849367 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" Oct 31 01:15:39.849601 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.849395 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br" event={"ID":"66b984db-1113-44af-b6d9-6cf452c97187","Type":"ContainerDied","Data":"ae091e58577972c45679de73f8cc337e8ef04d265cf5a3024203d4d68136cf6e"} Oct 31 01:15:39.851402 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.851377 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" event={"ID":"a2ef7da0-2a19-4794-a80f-9badb9461e9a","Type":"ContainerDied","Data":"40b3e1ae3ae7b637e60abd854227198bd015650732fdfcf4cdb1f25508f06a7c"} Oct 31 01:15:39.851637 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.851621 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz" Oct 31 01:15:39.856960 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.856938 4440 scope.go:122] "RemoveContainer" containerID="829a6af953a02de438e6a74379be137cd8a51cadc3ec4a685b782715a031d660" Oct 31 01:15:39.866167 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.866134 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw"] Oct 31 01:15:39.869676 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.869645 4440 scope.go:122] "RemoveContainer" containerID="8283803af9dca21d4b672094bf290e6bab2c020f091996b1a51930df9bdb16aa" Oct 31 01:15:39.870264 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.870241 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-b5rrw"] Oct 31 01:15:39.885992 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.885923 4440 scope.go:122] "RemoveContainer" containerID="29979f9137e8c116558b7b16cfac14685809a3092cad8481902db8d834367e07" Oct 31 01:15:39.888578 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.888527 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz"] Oct 31 01:15:39.891574 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.891549 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-xpfwz"] Oct 31 01:15:39.903864 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.903828 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz"] Oct 31 01:15:39.909729 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.909695 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-gbxrz"] Oct 31 01:15:39.932529 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.932485 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br"] Oct 31 01:15:39.935354 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:39.935324 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-487a6c-openshift-default-567f7cfc97-7b7br"] Oct 31 01:15:40.042342 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:40.042306 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:41.080435 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:41.080382 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:41.780487 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:41.780446 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="613c5424-a9e2-43da-95aa-807e2b99ab2e" path="/var/lib/kubelet/pods/613c5424-a9e2-43da-95aa-807e2b99ab2e/volumes" Oct 31 01:15:41.781013 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:41.780993 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66b984db-1113-44af-b6d9-6cf452c97187" path="/var/lib/kubelet/pods/66b984db-1113-44af-b6d9-6cf452c97187/volumes" Oct 31 01:15:41.781484 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:41.781467 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2ef7da0-2a19-4794-a80f-9badb9461e9a" path="/var/lib/kubelet/pods/a2ef7da0-2a19-4794-a80f-9badb9461e9a/volumes" Oct 31 01:15:41.782249 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:41.782229 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8555032-1556-4f20-a9bb-7f1dcbf99dcc" path="/var/lib/kubelet/pods/f8555032-1556-4f20-a9bb-7f1dcbf99dcc/volumes" Oct 31 01:15:46.673355 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:46.673314 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:46.773784 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:46.773741 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:56.396693 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.396642 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-c4977"] Oct 31 01:15:56.397155 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397075 4440 cpu_manager.go:414] "containerMap: removing container" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" Oct 31 01:15:56.397155 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397087 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" Oct 31 01:15:56.397155 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397102 4440 cpu_manager.go:414] "containerMap: removing container" podUID="613c5424-a9e2-43da-95aa-807e2b99ab2e" containerName="istio-proxy" Oct 31 01:15:56.397155 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397107 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="613c5424-a9e2-43da-95aa-807e2b99ab2e" containerName="istio-proxy" Oct 31 01:15:56.397155 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397119 4440 cpu_manager.go:414] "containerMap: removing container" podUID="cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf" containerName="istio-proxy" Oct 31 01:15:56.397155 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397124 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf" containerName="istio-proxy" Oct 31 01:15:56.397155 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397131 4440 cpu_manager.go:414] "containerMap: removing container" podUID="0a3e8a39-48ca-4063-93b8-8afa2f1f18ac" containerName="payload-processing" Oct 31 01:15:56.397155 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397136 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a3e8a39-48ca-4063-93b8-8afa2f1f18ac" containerName="payload-processing" Oct 31 01:15:56.397155 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397144 4440 cpu_manager.go:414] "containerMap: removing container" podUID="c7c55c28-c3ca-4832-9559-b05e99a3995d" containerName="cleanup" Oct 31 01:15:56.397155 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397149 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7c55c28-c3ca-4832-9559-b05e99a3995d" containerName="cleanup" Oct 31 01:15:56.397155 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397155 4440 cpu_manager.go:414] "containerMap: removing container" podUID="5f420497-f98d-4a0c-8552-fca5c3f002f2" containerName="istio-proxy" Oct 31 01:15:56.397155 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397159 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f420497-f98d-4a0c-8552-fca5c3f002f2" containerName="istio-proxy" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397167 4440 cpu_manager.go:414] "containerMap: removing container" podUID="c7c55c28-c3ca-4832-9559-b05e99a3995d" containerName="cleanup" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397172 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7c55c28-c3ca-4832-9559-b05e99a3995d" containerName="cleanup" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397181 4440 cpu_manager.go:414] "containerMap: removing container" podUID="f8555032-1556-4f20-a9bb-7f1dcbf99dcc" containerName="istio-proxy" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397186 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8555032-1556-4f20-a9bb-7f1dcbf99dcc" containerName="istio-proxy" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397193 4440 cpu_manager.go:414] "containerMap: removing container" podUID="7ee3eb69-94de-447c-b9ec-194a0458f2e7" containerName="istio-proxy" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397199 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ee3eb69-94de-447c-b9ec-194a0458f2e7" containerName="istio-proxy" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397207 4440 cpu_manager.go:414] "containerMap: removing container" podUID="e4160358-c655-4a6b-bb39-fc8677804241" containerName="payload-pre-processing" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397211 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4160358-c655-4a6b-bb39-fc8677804241" containerName="payload-pre-processing" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397218 4440 cpu_manager.go:414] "containerMap: removing container" podUID="66b984db-1113-44af-b6d9-6cf452c97187" containerName="istio-proxy" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397223 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66b984db-1113-44af-b6d9-6cf452c97187" containerName="istio-proxy" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397231 4440 cpu_manager.go:414] "containerMap: removing container" podUID="81095022-acb2-4b1c-acc6-a9501b0ea478" containerName="istio-proxy" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397236 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81095022-acb2-4b1c-acc6-a9501b0ea478" containerName="istio-proxy" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397244 4440 cpu_manager.go:414] "containerMap: removing container" podUID="3b4e5ddd-9933-47e1-9265-085858232375" containerName="istio-proxy" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397249 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b4e5ddd-9933-47e1-9265-085858232375" containerName="istio-proxy" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397255 4440 cpu_manager.go:414] "containerMap: removing container" podUID="823db35e-e37c-448d-a70d-96b5a218ac5e" containerName="istio-proxy" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397259 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="823db35e-e37c-448d-a70d-96b5a218ac5e" containerName="istio-proxy" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397267 4440 cpu_manager.go:414] "containerMap: removing container" podUID="f8555032-1556-4f20-a9bb-7f1dcbf99dcc" containerName="istio-proxy" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397272 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8555032-1556-4f20-a9bb-7f1dcbf99dcc" containerName="istio-proxy" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397278 4440 cpu_manager.go:414] "containerMap: removing container" podUID="a2ef7da0-2a19-4794-a80f-9badb9461e9a" containerName="istio-proxy" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397282 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2ef7da0-2a19-4794-a80f-9badb9461e9a" containerName="istio-proxy" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397290 4440 cpu_manager.go:414] "containerMap: removing container" podUID="877c89f5-20ff-4b40-a161-15b6e135c4b9" containerName="cleanup" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397294 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="877c89f5-20ff-4b40-a161-15b6e135c4b9" containerName="cleanup" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397301 4440 cpu_manager.go:414] "containerMap: removing container" podUID="877c89f5-20ff-4b40-a161-15b6e135c4b9" containerName="cleanup" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397305 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="877c89f5-20ff-4b40-a161-15b6e135c4b9" containerName="cleanup" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397313 4440 cpu_manager.go:414] "containerMap: removing container" podUID="117032f9-b742-432f-ac5f-70c9982b919f" containerName="istio-proxy" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397318 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="117032f9-b742-432f-ac5f-70c9982b919f" containerName="istio-proxy" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397324 4440 cpu_manager.go:414] "containerMap: removing container" podUID="b192b955-7018-4d48-b3c6-c58d86eeb5b2" containerName="istio-proxy" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397329 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b192b955-7018-4d48-b3c6-c58d86eeb5b2" containerName="istio-proxy" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397335 4440 cpu_manager.go:414] "containerMap: removing container" podUID="8f8c2432-9405-4f4f-b24f-1b73ea0da86a" containerName="istio-proxy" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397339 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f8c2432-9405-4f4f-b24f-1b73ea0da86a" containerName="istio-proxy" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397347 4440 cpu_manager.go:414] "containerMap: removing container" podUID="877c89f5-20ff-4b40-a161-15b6e135c4b9" containerName="cleanup" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397365 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="877c89f5-20ff-4b40-a161-15b6e135c4b9" containerName="cleanup" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397431 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7c55c28-c3ca-4832-9559-b05e99a3995d" containerName="cleanup" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397440 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="117032f9-b742-432f-ac5f-70c9982b919f" containerName="istio-proxy" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397445 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="b192b955-7018-4d48-b3c6-c58d86eeb5b2" containerName="istio-proxy" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397451 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f8c2432-9405-4f4f-b24f-1b73ea0da86a" containerName="istio-proxy" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397456 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf" containerName="istio-proxy" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397461 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="877c89f5-20ff-4b40-a161-15b6e135c4b9" containerName="cleanup" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397468 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="054f9e47-675f-4722-b743-79bbdc826585" containerName="main" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397474 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="613c5424-a9e2-43da-95aa-807e2b99ab2e" containerName="istio-proxy" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397481 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a3e8a39-48ca-4063-93b8-8afa2f1f18ac" containerName="payload-processing" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397485 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f420497-f98d-4a0c-8552-fca5c3f002f2" containerName="istio-proxy" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397491 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7c55c28-c3ca-4832-9559-b05e99a3995d" containerName="cleanup" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397497 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8555032-1556-4f20-a9bb-7f1dcbf99dcc" containerName="istio-proxy" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397503 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ee3eb69-94de-447c-b9ec-194a0458f2e7" containerName="istio-proxy" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397509 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4160358-c655-4a6b-bb39-fc8677804241" containerName="payload-pre-processing" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397515 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="66b984db-1113-44af-b6d9-6cf452c97187" containerName="istio-proxy" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397519 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="81095022-acb2-4b1c-acc6-a9501b0ea478" containerName="istio-proxy" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397525 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b4e5ddd-9933-47e1-9265-085858232375" containerName="istio-proxy" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397530 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="823db35e-e37c-448d-a70d-96b5a218ac5e" containerName="istio-proxy" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397534 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8555032-1556-4f20-a9bb-7f1dcbf99dcc" containerName="istio-proxy" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397539 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2ef7da0-2a19-4794-a80f-9badb9461e9a" containerName="istio-proxy" Oct 31 01:15:56.397580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.397545 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="877c89f5-20ff-4b40-a161-15b6e135c4b9" containerName="cleanup" Oct 31 01:15:56.400746 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.400723 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-c4977" Oct 31 01:15:56.405282 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.405251 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:15:56.405443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.405320 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:15:56.405523 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.405471 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:15:56.406360 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.406336 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zs6l6\"" Oct 31 01:15:56.410142 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.410114 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-c4977"] Oct 31 01:15:56.442414 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.442384 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23a3f91f-a4a2-4c88-89e9-8e13f50a5416-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-c4977\" (UID: \"23a3f91f-a4a2-4c88-89e9-8e13f50a5416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-c4977" Oct 31 01:15:56.442551 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.442462 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23a3f91f-a4a2-4c88-89e9-8e13f50a5416-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-c4977\" (UID: \"23a3f91f-a4a2-4c88-89e9-8e13f50a5416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-c4977" Oct 31 01:15:56.442618 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.442577 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff246\" (UniqueName: \"kubernetes.io/projected/23a3f91f-a4a2-4c88-89e9-8e13f50a5416-kube-api-access-ff246\") pod \"maas-api-revoke-sub-85237e45-c4977\" (UID: \"23a3f91f-a4a2-4c88-89e9-8e13f50a5416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-c4977" Oct 31 01:15:56.544269 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.544230 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23a3f91f-a4a2-4c88-89e9-8e13f50a5416-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-c4977\" (UID: \"23a3f91f-a4a2-4c88-89e9-8e13f50a5416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-c4977" Oct 31 01:15:56.544550 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.544298 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23a3f91f-a4a2-4c88-89e9-8e13f50a5416-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-c4977\" (UID: \"23a3f91f-a4a2-4c88-89e9-8e13f50a5416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-c4977" Oct 31 01:15:56.552894 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.544950 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff246\" (UniqueName: \"kubernetes.io/projected/23a3f91f-a4a2-4c88-89e9-8e13f50a5416-kube-api-access-ff246\") pod \"maas-api-revoke-sub-85237e45-c4977\" (UID: \"23a3f91f-a4a2-4c88-89e9-8e13f50a5416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-c4977" Oct 31 01:15:56.552894 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.545819 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23a3f91f-a4a2-4c88-89e9-8e13f50a5416-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-c4977\" (UID: \"23a3f91f-a4a2-4c88-89e9-8e13f50a5416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-c4977" Oct 31 01:15:56.556725 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.554169 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff246\" (UniqueName: \"kubernetes.io/projected/23a3f91f-a4a2-4c88-89e9-8e13f50a5416-kube-api-access-ff246\") pod \"maas-api-revoke-sub-85237e45-c4977\" (UID: \"23a3f91f-a4a2-4c88-89e9-8e13f50a5416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-c4977" Oct 31 01:15:56.560253 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.560222 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23a3f91f-a4a2-4c88-89e9-8e13f50a5416-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-c4977\" (UID: \"23a3f91f-a4a2-4c88-89e9-8e13f50a5416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-c4977" Oct 31 01:15:56.720514 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:56.720414 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-c4977" Oct 31 01:15:57.000467 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:57.000436 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-c4977"] Oct 31 01:15:57.001542 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:15:57.001507 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a3f91f_a4a2_4c88_89e9_8e13f50a5416.slice/crio-f5c1b18cfbab209e6223d1af1d77bdc30e701e7d178c632e1cfe6225cded7b62 WatchSource:0}: Error finding container f5c1b18cfbab209e6223d1af1d77bdc30e701e7d178c632e1cfe6225cded7b62: Status 404 returned error can't find the container with id f5c1b18cfbab209e6223d1af1d77bdc30e701e7d178c632e1cfe6225cded7b62 Oct 31 01:15:57.854154 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:57.854117 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:15:57.977467 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:57.977343 4440 generic.go:356] "Generic (PLEG): container finished" podID="23a3f91f-a4a2-4c88-89e9-8e13f50a5416" containerID="551aaeae72aa26c6ff19155a9eaafae0c75e2c4c4e87df0feaea40af01c3ff94" exitCode=0 Oct 31 01:15:57.977614 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:57.977458 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-c4977" event={"ID":"23a3f91f-a4a2-4c88-89e9-8e13f50a5416","Type":"ContainerDied","Data":"551aaeae72aa26c6ff19155a9eaafae0c75e2c4c4e87df0feaea40af01c3ff94"} Oct 31 01:15:57.977614 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:57.977516 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-c4977" event={"ID":"23a3f91f-a4a2-4c88-89e9-8e13f50a5416","Type":"ContainerStarted","Data":"f5c1b18cfbab209e6223d1af1d77bdc30e701e7d178c632e1cfe6225cded7b62"} Oct 31 01:15:59.132406 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:59.132375 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-c4977" Oct 31 01:15:59.173687 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:59.173651 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23a3f91f-a4a2-4c88-89e9-8e13f50a5416-kube-api-access-ff246\" (UniqueName: \"kubernetes.io/projected/23a3f91f-a4a2-4c88-89e9-8e13f50a5416-kube-api-access-ff246\") pod \"23a3f91f-a4a2-4c88-89e9-8e13f50a5416\" (UID: \"23a3f91f-a4a2-4c88-89e9-8e13f50a5416\") " Oct 31 01:15:59.173884 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:59.173798 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23a3f91f-a4a2-4c88-89e9-8e13f50a5416-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23a3f91f-a4a2-4c88-89e9-8e13f50a5416-maas-api-openshift-ca\") pod \"23a3f91f-a4a2-4c88-89e9-8e13f50a5416\" (UID: \"23a3f91f-a4a2-4c88-89e9-8e13f50a5416\") " Oct 31 01:15:59.173884 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:59.173844 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23a3f91f-a4a2-4c88-89e9-8e13f50a5416-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23a3f91f-a4a2-4c88-89e9-8e13f50a5416-maas-api-xks-ca\") pod \"23a3f91f-a4a2-4c88-89e9-8e13f50a5416\" (UID: \"23a3f91f-a4a2-4c88-89e9-8e13f50a5416\") " Oct 31 01:15:59.174266 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:59.174228 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23a3f91f-a4a2-4c88-89e9-8e13f50a5416-maas-api-openshift-ca" pod "23a3f91f-a4a2-4c88-89e9-8e13f50a5416" (UID: "23a3f91f-a4a2-4c88-89e9-8e13f50a5416"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:59.175996 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:59.175962 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23a3f91f-a4a2-4c88-89e9-8e13f50a5416-maas-api-xks-ca" pod "23a3f91f-a4a2-4c88-89e9-8e13f50a5416" (UID: "23a3f91f-a4a2-4c88-89e9-8e13f50a5416"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:15:59.176104 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:59.176042 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a3f91f-a4a2-4c88-89e9-8e13f50a5416-kube-api-access-ff246" pod "23a3f91f-a4a2-4c88-89e9-8e13f50a5416" (UID: "23a3f91f-a4a2-4c88-89e9-8e13f50a5416"). InnerVolumeSpecName "kube-api-access-ff246". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:59.275654 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:59.275543 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff246\" (UniqueName: \"kubernetes.io/projected/23a3f91f-a4a2-4c88-89e9-8e13f50a5416-kube-api-access-ff246\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:59.275654 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:59.275575 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23a3f91f-a4a2-4c88-89e9-8e13f50a5416-maas-api-openshift-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:59.275654 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:59.275587 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23a3f91f-a4a2-4c88-89e9-8e13f50a5416-maas-api-xks-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:59.990382 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:59.990352 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-c4977" Oct 31 01:15:59.990593 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:59.990374 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-c4977" event={"ID":"23a3f91f-a4a2-4c88-89e9-8e13f50a5416","Type":"ContainerDied","Data":"f5c1b18cfbab209e6223d1af1d77bdc30e701e7d178c632e1cfe6225cded7b62"} Oct 31 01:15:59.990593 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:15:59.990435 4440 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5c1b18cfbab209e6223d1af1d77bdc30e701e7d178c632e1cfe6225cded7b62" Oct 31 01:16:01.006494 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:16:01.006456 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:16:07.960432 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:07.960309 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:09.865245 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:09.865191 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:30.999535 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:16:30.999490 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:16:44.766182 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:44.766145 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:16:46.804443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:46.804075 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:49.639224 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:49.639186 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dsqxl"] Oct 31 01:16:49.639658 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:49.639644 4440 cpu_manager.go:414] "containerMap: removing container" podUID="cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf" containerName="istio-proxy" Oct 31 01:16:49.639658 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:49.639658 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf" containerName="istio-proxy" Oct 31 01:16:49.639733 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:49.639670 4440 cpu_manager.go:414] "containerMap: removing container" podUID="c7c55c28-c3ca-4832-9559-b05e99a3995d" containerName="cleanup" Oct 31 01:16:49.639733 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:49.639675 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7c55c28-c3ca-4832-9559-b05e99a3995d" containerName="cleanup" Oct 31 01:16:49.639733 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:49.639701 4440 cpu_manager.go:414] "containerMap: removing container" podUID="23a3f91f-a4a2-4c88-89e9-8e13f50a5416" containerName="revoke-keys" Oct 31 01:16:49.639733 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:49.639706 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23a3f91f-a4a2-4c88-89e9-8e13f50a5416" containerName="revoke-keys" Oct 31 01:16:49.639857 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:49.639768 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7c55c28-c3ca-4832-9559-b05e99a3995d" containerName="cleanup" Oct 31 01:16:49.639857 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:49.639782 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="23a3f91f-a4a2-4c88-89e9-8e13f50a5416" containerName="revoke-keys" Oct 31 01:16:49.639857 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:49.639788 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd3cc4c7-0978-44a4-bec3-cb1d3bd89daf" containerName="istio-proxy" Oct 31 01:16:49.639857 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:49.639794 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="877c89f5-20ff-4b40-a161-15b6e135c4b9" containerName="cleanup" Oct 31 01:16:49.643042 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:49.643020 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dsqxl" Oct 31 01:16:49.647069 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:49.647043 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:16:49.647231 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:49.647213 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zs6l6\"" Oct 31 01:16:49.647932 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:49.647315 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:16:49.647932 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:49.647315 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:16:49.650391 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:49.650366 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dsqxl"] Oct 31 01:16:49.761221 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:49.761182 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54448a0d-f363-455a-bd78-7026acfc2401-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-dsqxl\" (UID: \"54448a0d-f363-455a-bd78-7026acfc2401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dsqxl" Oct 31 01:16:49.761413 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:49.761300 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9znbm\" (UniqueName: \"kubernetes.io/projected/54448a0d-f363-455a-bd78-7026acfc2401-kube-api-access-9znbm\") pod \"maas-api-revoke-sub-c1e271a9-dsqxl\" (UID: \"54448a0d-f363-455a-bd78-7026acfc2401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dsqxl" Oct 31 01:16:49.761413 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:49.761378 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54448a0d-f363-455a-bd78-7026acfc2401-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-dsqxl\" (UID: \"54448a0d-f363-455a-bd78-7026acfc2401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dsqxl" Oct 31 01:16:49.862177 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:49.862136 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9znbm\" (UniqueName: \"kubernetes.io/projected/54448a0d-f363-455a-bd78-7026acfc2401-kube-api-access-9znbm\") pod \"maas-api-revoke-sub-c1e271a9-dsqxl\" (UID: \"54448a0d-f363-455a-bd78-7026acfc2401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dsqxl" Oct 31 01:16:49.862372 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:49.862226 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54448a0d-f363-455a-bd78-7026acfc2401-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-dsqxl\" (UID: \"54448a0d-f363-455a-bd78-7026acfc2401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dsqxl" Oct 31 01:16:49.862372 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:49.862285 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54448a0d-f363-455a-bd78-7026acfc2401-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-dsqxl\" (UID: \"54448a0d-f363-455a-bd78-7026acfc2401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dsqxl" Oct 31 01:16:49.862913 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:49.862892 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54448a0d-f363-455a-bd78-7026acfc2401-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-dsqxl\" (UID: \"54448a0d-f363-455a-bd78-7026acfc2401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dsqxl" Oct 31 01:16:49.864658 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:49.864634 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54448a0d-f363-455a-bd78-7026acfc2401-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-dsqxl\" (UID: \"54448a0d-f363-455a-bd78-7026acfc2401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dsqxl" Oct 31 01:16:49.869567 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:49.869540 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9znbm\" (UniqueName: \"kubernetes.io/projected/54448a0d-f363-455a-bd78-7026acfc2401-kube-api-access-9znbm\") pod \"maas-api-revoke-sub-c1e271a9-dsqxl\" (UID: \"54448a0d-f363-455a-bd78-7026acfc2401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dsqxl" Oct 31 01:16:49.963034 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:49.962992 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dsqxl" Oct 31 01:16:50.238806 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:50.238773 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dsqxl"] Oct 31 01:16:50.239987 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:16:50.239963 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54448a0d_f363_455a_bd78_7026acfc2401.slice/crio-545d43912dba9d8199285353e2ea60ce71215c18edfab7241af0c28e8e2b9508 WatchSource:0}: Error finding container 545d43912dba9d8199285353e2ea60ce71215c18edfab7241af0c28e8e2b9508: Status 404 returned error can't find the container with id 545d43912dba9d8199285353e2ea60ce71215c18edfab7241af0c28e8e2b9508 Oct 31 01:16:50.330703 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:50.330667 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dsqxl" event={"ID":"54448a0d-f363-455a-bd78-7026acfc2401","Type":"ContainerStarted","Data":"545d43912dba9d8199285353e2ea60ce71215c18edfab7241af0c28e8e2b9508"} Oct 31 01:16:51.339000 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:51.338965 4440 generic.go:356] "Generic (PLEG): container finished" podID="54448a0d-f363-455a-bd78-7026acfc2401" containerID="d7193ceae87970ebee59011b63672f948579f177638e4272b820d7db25b0054c" exitCode=0 Oct 31 01:16:51.339396 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:51.339020 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dsqxl" event={"ID":"54448a0d-f363-455a-bd78-7026acfc2401","Type":"ContainerDied","Data":"d7193ceae87970ebee59011b63672f948579f177638e4272b820d7db25b0054c"} Oct 31 01:16:52.497585 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:52.497559 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dsqxl" Oct 31 01:16:52.586209 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:52.586174 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54448a0d-f363-455a-bd78-7026acfc2401-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54448a0d-f363-455a-bd78-7026acfc2401-maas-api-xks-ca\") pod \"54448a0d-f363-455a-bd78-7026acfc2401\" (UID: \"54448a0d-f363-455a-bd78-7026acfc2401\") " Oct 31 01:16:52.586209 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:52.586213 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54448a0d-f363-455a-bd78-7026acfc2401-kube-api-access-9znbm\" (UniqueName: \"kubernetes.io/projected/54448a0d-f363-455a-bd78-7026acfc2401-kube-api-access-9znbm\") pod \"54448a0d-f363-455a-bd78-7026acfc2401\" (UID: \"54448a0d-f363-455a-bd78-7026acfc2401\") " Oct 31 01:16:52.586505 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:52.586261 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54448a0d-f363-455a-bd78-7026acfc2401-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54448a0d-f363-455a-bd78-7026acfc2401-maas-api-openshift-ca\") pod \"54448a0d-f363-455a-bd78-7026acfc2401\" (UID: \"54448a0d-f363-455a-bd78-7026acfc2401\") " Oct 31 01:16:52.586707 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:52.586684 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54448a0d-f363-455a-bd78-7026acfc2401-maas-api-openshift-ca" pod "54448a0d-f363-455a-bd78-7026acfc2401" (UID: "54448a0d-f363-455a-bd78-7026acfc2401"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:16:52.588272 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:52.588242 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54448a0d-f363-455a-bd78-7026acfc2401-maas-api-xks-ca" pod "54448a0d-f363-455a-bd78-7026acfc2401" (UID: "54448a0d-f363-455a-bd78-7026acfc2401"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:16:52.588365 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:52.588330 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54448a0d-f363-455a-bd78-7026acfc2401-kube-api-access-9znbm" pod "54448a0d-f363-455a-bd78-7026acfc2401" (UID: "54448a0d-f363-455a-bd78-7026acfc2401"). InnerVolumeSpecName "kube-api-access-9znbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:16:52.688075 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:52.688036 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54448a0d-f363-455a-bd78-7026acfc2401-maas-api-openshift-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:16:52.688269 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:52.688086 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54448a0d-f363-455a-bd78-7026acfc2401-maas-api-xks-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:16:52.688269 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:52.688101 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9znbm\" (UniqueName: \"kubernetes.io/projected/54448a0d-f363-455a-bd78-7026acfc2401-kube-api-access-9znbm\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:16:53.352910 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:53.352880 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dsqxl" Oct 31 01:16:53.353099 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:53.352913 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dsqxl" event={"ID":"54448a0d-f363-455a-bd78-7026acfc2401","Type":"ContainerDied","Data":"545d43912dba9d8199285353e2ea60ce71215c18edfab7241af0c28e8e2b9508"} Oct 31 01:16:53.353099 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:53.352966 4440 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="545d43912dba9d8199285353e2ea60ce71215c18edfab7241af0c28e8e2b9508" Oct 31 01:16:56.524897 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:16:56.524804 4440 scope.go:122] "RemoveContainer" containerID="a490227f3a04ca24e002491fed82910c318ffc6d73a7bb60a390977ae0159cb2" Oct 31 01:17:00.478567 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:00.478524 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:17:01.005979 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:17:01.005870 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:17:04.714451 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:04.713277 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:17:08.431180 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:08.431136 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g9f7d"] Oct 31 01:17:08.431773 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:08.431750 4440 cpu_manager.go:414] "containerMap: removing container" podUID="54448a0d-f363-455a-bd78-7026acfc2401" containerName="revoke-keys" Oct 31 01:17:08.431773 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:08.431773 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54448a0d-f363-455a-bd78-7026acfc2401" containerName="revoke-keys" Oct 31 01:17:08.431891 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:08.431883 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="54448a0d-f363-455a-bd78-7026acfc2401" containerName="revoke-keys" Oct 31 01:17:08.436590 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:08.436565 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g9f7d" Oct 31 01:17:08.441039 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:08.441009 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:17:08.441330 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:08.441297 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:17:08.441330 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:08.441298 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zs6l6\"" Oct 31 01:17:08.441540 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:08.441496 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:17:08.442720 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:08.442696 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g9f7d"] Oct 31 01:17:08.464203 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:08.464152 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-g9f7d\" (UID: \"e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g9f7d" Oct 31 01:17:08.464368 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:08.464218 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj6m7\" (UniqueName: \"kubernetes.io/projected/e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2-kube-api-access-hj6m7\") pod \"maas-api-revoke-sub-2e1f5b5f-g9f7d\" (UID: \"e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g9f7d" Oct 31 01:17:08.464368 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:08.464335 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-g9f7d\" (UID: \"e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g9f7d" Oct 31 01:17:08.566114 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:08.566067 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-g9f7d\" (UID: \"e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g9f7d" Oct 31 01:17:08.566342 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:08.566131 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj6m7\" (UniqueName: \"kubernetes.io/projected/e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2-kube-api-access-hj6m7\") pod \"maas-api-revoke-sub-2e1f5b5f-g9f7d\" (UID: \"e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g9f7d" Oct 31 01:17:08.566342 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:08.566173 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-g9f7d\" (UID: \"e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g9f7d" Oct 31 01:17:08.566963 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:08.566942 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-g9f7d\" (UID: \"e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g9f7d" Oct 31 01:17:08.568611 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:08.568594 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-g9f7d\" (UID: \"e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g9f7d" Oct 31 01:17:08.573566 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:08.573535 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj6m7\" (UniqueName: \"kubernetes.io/projected/e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2-kube-api-access-hj6m7\") pod \"maas-api-revoke-sub-2e1f5b5f-g9f7d\" (UID: \"e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g9f7d" Oct 31 01:17:08.753478 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:08.753375 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g9f7d" Oct 31 01:17:09.033066 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:09.033032 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g9f7d"] Oct 31 01:17:09.034195 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:17:09.034161 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode367eb3c_bee2_45ee_9b8f_1d3bb4f78ed2.slice/crio-56c36378ca0c47b354d2d76e07c6c4a97e3c0e360eba52529a55dc20806cd7bd WatchSource:0}: Error finding container 56c36378ca0c47b354d2d76e07c6c4a97e3c0e360eba52529a55dc20806cd7bd: Status 404 returned error can't find the container with id 56c36378ca0c47b354d2d76e07c6c4a97e3c0e360eba52529a55dc20806cd7bd Oct 31 01:17:09.566279 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:09.566184 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g9f7d" event={"ID":"e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2","Type":"ContainerStarted","Data":"79a9578a68d2ccce4914caa27c4a514447d1360c248b54c794e7f4bdc8a9e538"} Oct 31 01:17:09.566279 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:09.566248 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g9f7d" event={"ID":"e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2","Type":"ContainerStarted","Data":"56c36378ca0c47b354d2d76e07c6c4a97e3c0e360eba52529a55dc20806cd7bd"} Oct 31 01:17:09.583535 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:09.583472 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g9f7d" podStartSLOduration=1.583455662 podStartE2EDuration="1.583455662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:17:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:17:09.58221264 +0000 UTC m=+1273.940934959" watchObservedRunningTime="2026-10-31 01:17:09.583455662 +0000 UTC m=+1273.942177981" Oct 31 01:17:10.573691 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:10.573654 4440 generic.go:356] "Generic (PLEG): container finished" podID="e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2" containerID="79a9578a68d2ccce4914caa27c4a514447d1360c248b54c794e7f4bdc8a9e538" exitCode=0 Oct 31 01:17:10.574099 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:10.573742 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g9f7d" event={"ID":"e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2","Type":"ContainerDied","Data":"79a9578a68d2ccce4914caa27c4a514447d1360c248b54c794e7f4bdc8a9e538"} Oct 31 01:17:11.724961 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:11.724928 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g9f7d" Oct 31 01:17:11.800975 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:11.800931 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2-maas-api-xks-ca\") pod \"e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2\" (UID: \"e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2\") " Oct 31 01:17:11.801184 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:11.800988 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2-kube-api-access-hj6m7\" (UniqueName: \"kubernetes.io/projected/e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2-kube-api-access-hj6m7\") pod \"e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2\" (UID: \"e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2\") " Oct 31 01:17:11.801184 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:11.801045 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2-maas-api-openshift-ca\") pod \"e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2\" (UID: \"e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2\") " Oct 31 01:17:11.801457 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:11.801395 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2-maas-api-openshift-ca" pod "e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2" (UID: "e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:17:11.801986 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:11.801959 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2-maas-api-openshift-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:17:11.803259 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:11.803231 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2-maas-api-xks-ca" pod "e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2" (UID: "e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:17:11.803350 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:11.803293 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2-kube-api-access-hj6m7" pod "e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2" (UID: "e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2"). InnerVolumeSpecName "kube-api-access-hj6m7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:17:11.903426 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:11.903327 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2-maas-api-xks-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:17:11.903426 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:11.903363 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj6m7\" (UniqueName: \"kubernetes.io/projected/e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2-kube-api-access-hj6m7\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:17:12.586190 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:12.586159 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g9f7d" Oct 31 01:17:12.586373 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:12.586203 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g9f7d" event={"ID":"e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2","Type":"ContainerDied","Data":"56c36378ca0c47b354d2d76e07c6c4a97e3c0e360eba52529a55dc20806cd7bd"} Oct 31 01:17:12.586373 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:12.586257 4440 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56c36378ca0c47b354d2d76e07c6c4a97e3c0e360eba52529a55dc20806cd7bd" Oct 31 01:17:14.598734 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:14.598698 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:14.704660 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:14.704626 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:19.368611 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:19.368568 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:17:19.398154 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:19.398110 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dql8q"] Oct 31 01:17:19.398605 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:19.398588 4440 cpu_manager.go:414] "containerMap: removing container" podUID="e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2" containerName="revoke-keys" Oct 31 01:17:19.398605 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:19.398605 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2" containerName="revoke-keys" Oct 31 01:17:19.398752 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:19.398686 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2" containerName="revoke-keys" Oct 31 01:17:19.402085 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:19.402062 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dql8q" Oct 31 01:17:19.408457 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:19.408411 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:17:19.408841 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:19.408820 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zs6l6\"" Oct 31 01:17:19.408969 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:19.408870 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:17:19.408969 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:19.408946 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:17:19.411661 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:19.411592 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dql8q"] Oct 31 01:17:19.483178 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:19.483138 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk26s\" (UniqueName: \"kubernetes.io/projected/ef1031f7-24de-4eb3-8d23-eb2db2be80fe-kube-api-access-nk26s\") pod \"maas-api-revoke-sub-ec843018-dql8q\" (UID: \"ef1031f7-24de-4eb3-8d23-eb2db2be80fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dql8q" Oct 31 01:17:19.483370 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:19.483253 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef1031f7-24de-4eb3-8d23-eb2db2be80fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-dql8q\" (UID: \"ef1031f7-24de-4eb3-8d23-eb2db2be80fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dql8q" Oct 31 01:17:19.483487 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:19.483459 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef1031f7-24de-4eb3-8d23-eb2db2be80fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-dql8q\" (UID: \"ef1031f7-24de-4eb3-8d23-eb2db2be80fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dql8q" Oct 31 01:17:19.584340 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:19.584292 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef1031f7-24de-4eb3-8d23-eb2db2be80fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-dql8q\" (UID: \"ef1031f7-24de-4eb3-8d23-eb2db2be80fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dql8q" Oct 31 01:17:19.584530 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:19.584405 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk26s\" (UniqueName: \"kubernetes.io/projected/ef1031f7-24de-4eb3-8d23-eb2db2be80fe-kube-api-access-nk26s\") pod \"maas-api-revoke-sub-ec843018-dql8q\" (UID: \"ef1031f7-24de-4eb3-8d23-eb2db2be80fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dql8q" Oct 31 01:17:19.584530 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:19.584514 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef1031f7-24de-4eb3-8d23-eb2db2be80fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-dql8q\" (UID: \"ef1031f7-24de-4eb3-8d23-eb2db2be80fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dql8q" Oct 31 01:17:19.585126 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:19.585096 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef1031f7-24de-4eb3-8d23-eb2db2be80fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-dql8q\" (UID: \"ef1031f7-24de-4eb3-8d23-eb2db2be80fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dql8q" Oct 31 01:17:19.586920 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:19.586888 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef1031f7-24de-4eb3-8d23-eb2db2be80fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-dql8q\" (UID: \"ef1031f7-24de-4eb3-8d23-eb2db2be80fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dql8q" Oct 31 01:17:19.592455 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:19.592399 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk26s\" (UniqueName: \"kubernetes.io/projected/ef1031f7-24de-4eb3-8d23-eb2db2be80fe-kube-api-access-nk26s\") pod \"maas-api-revoke-sub-ec843018-dql8q\" (UID: \"ef1031f7-24de-4eb3-8d23-eb2db2be80fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dql8q" Oct 31 01:17:19.726083 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:19.726049 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dql8q" Oct 31 01:17:20.042835 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:20.042800 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dql8q"] Oct 31 01:17:20.043728 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:17:20.043700 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef1031f7_24de_4eb3_8d23_eb2db2be80fe.slice/crio-f7b8d048db9af821a670a9421fec8e91f0076301ef946068b156ec117b41b869 WatchSource:0}: Error finding container f7b8d048db9af821a670a9421fec8e91f0076301ef946068b156ec117b41b869: Status 404 returned error can't find the container with id f7b8d048db9af821a670a9421fec8e91f0076301ef946068b156ec117b41b869 Oct 31 01:17:20.641047 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:20.640945 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dql8q" event={"ID":"ef1031f7-24de-4eb3-8d23-eb2db2be80fe","Type":"ContainerStarted","Data":"28ac6ddb0c8dddb7ea6999f15568465e1bfb9d59cee83572ff2dfa1cef6baca6"} Oct 31 01:17:20.641047 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:20.641003 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dql8q" event={"ID":"ef1031f7-24de-4eb3-8d23-eb2db2be80fe","Type":"ContainerStarted","Data":"f7b8d048db9af821a670a9421fec8e91f0076301ef946068b156ec117b41b869"} Oct 31 01:17:20.659604 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:20.659533 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dql8q" podStartSLOduration=1.6595135079999999 podStartE2EDuration="1.659513508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:17:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:17:20.657957757 +0000 UTC m=+1285.016680073" watchObservedRunningTime="2026-10-31 01:17:20.659513508 +0000 UTC m=+1285.018235826" Oct 31 01:17:21.648595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:21.648555 4440 generic.go:356] "Generic (PLEG): container finished" podID="ef1031f7-24de-4eb3-8d23-eb2db2be80fe" containerID="28ac6ddb0c8dddb7ea6999f15568465e1bfb9d59cee83572ff2dfa1cef6baca6" exitCode=0 Oct 31 01:17:21.649072 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:21.648607 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dql8q" event={"ID":"ef1031f7-24de-4eb3-8d23-eb2db2be80fe","Type":"ContainerDied","Data":"28ac6ddb0c8dddb7ea6999f15568465e1bfb9d59cee83572ff2dfa1cef6baca6"} Oct 31 01:17:23.040096 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:23.040070 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dql8q" Oct 31 01:17:23.129905 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:23.129867 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef1031f7-24de-4eb3-8d23-eb2db2be80fe-kube-api-access-nk26s\" (UniqueName: \"kubernetes.io/projected/ef1031f7-24de-4eb3-8d23-eb2db2be80fe-kube-api-access-nk26s\") pod \"ef1031f7-24de-4eb3-8d23-eb2db2be80fe\" (UID: \"ef1031f7-24de-4eb3-8d23-eb2db2be80fe\") " Oct 31 01:17:23.130307 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:23.130274 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef1031f7-24de-4eb3-8d23-eb2db2be80fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef1031f7-24de-4eb3-8d23-eb2db2be80fe-maas-api-openshift-ca\") pod \"ef1031f7-24de-4eb3-8d23-eb2db2be80fe\" (UID: \"ef1031f7-24de-4eb3-8d23-eb2db2be80fe\") " Oct 31 01:17:23.130637 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:23.130615 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef1031f7-24de-4eb3-8d23-eb2db2be80fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef1031f7-24de-4eb3-8d23-eb2db2be80fe-maas-api-xks-ca\") pod \"ef1031f7-24de-4eb3-8d23-eb2db2be80fe\" (UID: \"ef1031f7-24de-4eb3-8d23-eb2db2be80fe\") " Oct 31 01:17:23.139794 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:23.139764 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef1031f7-24de-4eb3-8d23-eb2db2be80fe-maas-api-openshift-ca" pod "ef1031f7-24de-4eb3-8d23-eb2db2be80fe" (UID: "ef1031f7-24de-4eb3-8d23-eb2db2be80fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:17:23.187135 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:23.187099 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef1031f7-24de-4eb3-8d23-eb2db2be80fe-maas-api-xks-ca" pod "ef1031f7-24de-4eb3-8d23-eb2db2be80fe" (UID: "ef1031f7-24de-4eb3-8d23-eb2db2be80fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:17:23.221514 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:23.221462 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef1031f7-24de-4eb3-8d23-eb2db2be80fe-kube-api-access-nk26s" pod "ef1031f7-24de-4eb3-8d23-eb2db2be80fe" (UID: "ef1031f7-24de-4eb3-8d23-eb2db2be80fe"). InnerVolumeSpecName "kube-api-access-nk26s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:17:23.239527 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:23.235168 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef1031f7-24de-4eb3-8d23-eb2db2be80fe-maas-api-openshift-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:17:23.239527 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:23.235215 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef1031f7-24de-4eb3-8d23-eb2db2be80fe-maas-api-xks-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:17:23.239527 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:23.235251 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk26s\" (UniqueName: \"kubernetes.io/projected/ef1031f7-24de-4eb3-8d23-eb2db2be80fe-kube-api-access-nk26s\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:17:23.584816 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:23.584718 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:23.722338 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:23.722298 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dql8q" event={"ID":"ef1031f7-24de-4eb3-8d23-eb2db2be80fe","Type":"ContainerDied","Data":"f7b8d048db9af821a670a9421fec8e91f0076301ef946068b156ec117b41b869"} Oct 31 01:17:23.722646 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:23.722623 4440 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7b8d048db9af821a670a9421fec8e91f0076301ef946068b156ec117b41b869" Oct 31 01:17:23.722837 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:23.722822 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dql8q" Oct 31 01:17:31.004336 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:17:31.004278 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:17:32.058369 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:32.058321 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:17:45.652241 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:45.652151 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dbvcx"] Oct 31 01:17:45.652686 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:45.652669 4440 cpu_manager.go:414] "containerMap: removing container" podUID="ef1031f7-24de-4eb3-8d23-eb2db2be80fe" containerName="revoke-keys" Oct 31 01:17:45.652731 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:45.652687 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef1031f7-24de-4eb3-8d23-eb2db2be80fe" containerName="revoke-keys" Oct 31 01:17:45.652783 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:45.652774 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef1031f7-24de-4eb3-8d23-eb2db2be80fe" containerName="revoke-keys" Oct 31 01:17:45.656074 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:45.656054 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dbvcx" Oct 31 01:17:45.663169 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:45.663135 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dbvcx"] Oct 31 01:17:45.664077 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:45.663690 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:17:45.664246 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:45.663716 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:17:45.664246 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:45.663762 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:17:45.664365 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:45.663800 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zs6l6\"" Oct 31 01:17:45.758747 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:45.758712 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptn5b\" (UniqueName: \"kubernetes.io/projected/33be65c2-e4d4-4fd8-bf80-1c3d9be57bee-kube-api-access-ptn5b\") pod \"maas-api-revoke-sub-046e1343-dbvcx\" (UID: \"33be65c2-e4d4-4fd8-bf80-1c3d9be57bee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dbvcx" Oct 31 01:17:45.758747 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:45.758752 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33be65c2-e4d4-4fd8-bf80-1c3d9be57bee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-dbvcx\" (UID: \"33be65c2-e4d4-4fd8-bf80-1c3d9be57bee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dbvcx" Oct 31 01:17:45.758990 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:45.758787 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33be65c2-e4d4-4fd8-bf80-1c3d9be57bee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-dbvcx\" (UID: \"33be65c2-e4d4-4fd8-bf80-1c3d9be57bee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dbvcx" Oct 31 01:17:45.860514 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:45.860472 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptn5b\" (UniqueName: \"kubernetes.io/projected/33be65c2-e4d4-4fd8-bf80-1c3d9be57bee-kube-api-access-ptn5b\") pod \"maas-api-revoke-sub-046e1343-dbvcx\" (UID: \"33be65c2-e4d4-4fd8-bf80-1c3d9be57bee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dbvcx" Oct 31 01:17:45.860514 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:45.860517 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33be65c2-e4d4-4fd8-bf80-1c3d9be57bee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-dbvcx\" (UID: \"33be65c2-e4d4-4fd8-bf80-1c3d9be57bee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dbvcx" Oct 31 01:17:45.860806 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:45.860553 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33be65c2-e4d4-4fd8-bf80-1c3d9be57bee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-dbvcx\" (UID: \"33be65c2-e4d4-4fd8-bf80-1c3d9be57bee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dbvcx" Oct 31 01:17:45.861270 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:45.861241 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33be65c2-e4d4-4fd8-bf80-1c3d9be57bee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-dbvcx\" (UID: \"33be65c2-e4d4-4fd8-bf80-1c3d9be57bee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dbvcx" Oct 31 01:17:45.863095 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:45.863074 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33be65c2-e4d4-4fd8-bf80-1c3d9be57bee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-dbvcx\" (UID: \"33be65c2-e4d4-4fd8-bf80-1c3d9be57bee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dbvcx" Oct 31 01:17:45.868991 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:45.868955 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptn5b\" (UniqueName: \"kubernetes.io/projected/33be65c2-e4d4-4fd8-bf80-1c3d9be57bee-kube-api-access-ptn5b\") pod \"maas-api-revoke-sub-046e1343-dbvcx\" (UID: \"33be65c2-e4d4-4fd8-bf80-1c3d9be57bee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dbvcx" Oct 31 01:17:45.978823 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:45.978783 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dbvcx" Oct 31 01:17:46.259923 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:46.259888 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dbvcx"] Oct 31 01:17:46.261116 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:17:46.261089 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33be65c2_e4d4_4fd8_bf80_1c3d9be57bee.slice/crio-dfb5f81ce726830228a41726d33044b98cf710fc0c166dbdf461d6bc79fd4774 WatchSource:0}: Error finding container dfb5f81ce726830228a41726d33044b98cf710fc0c166dbdf461d6bc79fd4774: Status 404 returned error can't find the container with id dfb5f81ce726830228a41726d33044b98cf710fc0c166dbdf461d6bc79fd4774 Oct 31 01:17:46.793267 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:46.793221 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:46.928340 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:46.928310 4440 generic.go:356] "Generic (PLEG): container finished" podID="33be65c2-e4d4-4fd8-bf80-1c3d9be57bee" containerID="e4628487e59a5b221c4cacdc3962848d57b68e1e407927dfa7ed41fe88e55768" exitCode=0 Oct 31 01:17:46.928600 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:46.928403 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dbvcx" event={"ID":"33be65c2-e4d4-4fd8-bf80-1c3d9be57bee","Type":"ContainerDied","Data":"e4628487e59a5b221c4cacdc3962848d57b68e1e407927dfa7ed41fe88e55768"} Oct 31 01:17:46.928600 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:46.928489 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dbvcx" event={"ID":"33be65c2-e4d4-4fd8-bf80-1c3d9be57bee","Type":"ContainerStarted","Data":"dfb5f81ce726830228a41726d33044b98cf710fc0c166dbdf461d6bc79fd4774"} Oct 31 01:17:48.078606 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:48.078579 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dbvcx" Oct 31 01:17:48.183777 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:48.183730 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33be65c2-e4d4-4fd8-bf80-1c3d9be57bee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33be65c2-e4d4-4fd8-bf80-1c3d9be57bee-maas-api-xks-ca\") pod \"33be65c2-e4d4-4fd8-bf80-1c3d9be57bee\" (UID: \"33be65c2-e4d4-4fd8-bf80-1c3d9be57bee\") " Oct 31 01:17:48.183967 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:48.183816 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33be65c2-e4d4-4fd8-bf80-1c3d9be57bee-kube-api-access-ptn5b\" (UniqueName: \"kubernetes.io/projected/33be65c2-e4d4-4fd8-bf80-1c3d9be57bee-kube-api-access-ptn5b\") pod \"33be65c2-e4d4-4fd8-bf80-1c3d9be57bee\" (UID: \"33be65c2-e4d4-4fd8-bf80-1c3d9be57bee\") " Oct 31 01:17:48.183967 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:48.183858 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33be65c2-e4d4-4fd8-bf80-1c3d9be57bee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33be65c2-e4d4-4fd8-bf80-1c3d9be57bee-maas-api-openshift-ca\") pod \"33be65c2-e4d4-4fd8-bf80-1c3d9be57bee\" (UID: \"33be65c2-e4d4-4fd8-bf80-1c3d9be57bee\") " Oct 31 01:17:48.184293 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:48.184264 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33be65c2-e4d4-4fd8-bf80-1c3d9be57bee-maas-api-openshift-ca" pod "33be65c2-e4d4-4fd8-bf80-1c3d9be57bee" (UID: "33be65c2-e4d4-4fd8-bf80-1c3d9be57bee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:17:48.186059 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:48.185997 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33be65c2-e4d4-4fd8-bf80-1c3d9be57bee-maas-api-xks-ca" pod "33be65c2-e4d4-4fd8-bf80-1c3d9be57bee" (UID: "33be65c2-e4d4-4fd8-bf80-1c3d9be57bee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:17:48.186059 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:48.186004 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33be65c2-e4d4-4fd8-bf80-1c3d9be57bee-kube-api-access-ptn5b" pod "33be65c2-e4d4-4fd8-bf80-1c3d9be57bee" (UID: "33be65c2-e4d4-4fd8-bf80-1c3d9be57bee"). InnerVolumeSpecName "kube-api-access-ptn5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:17:48.285441 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:48.285392 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33be65c2-e4d4-4fd8-bf80-1c3d9be57bee-maas-api-xks-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:17:48.285629 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:48.285470 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptn5b\" (UniqueName: \"kubernetes.io/projected/33be65c2-e4d4-4fd8-bf80-1c3d9be57bee-kube-api-access-ptn5b\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:17:48.285629 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:48.285482 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33be65c2-e4d4-4fd8-bf80-1c3d9be57bee-maas-api-openshift-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:17:48.941929 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:48.941888 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dbvcx" Oct 31 01:17:48.942129 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:48.941925 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dbvcx" event={"ID":"33be65c2-e4d4-4fd8-bf80-1c3d9be57bee","Type":"ContainerDied","Data":"dfb5f81ce726830228a41726d33044b98cf710fc0c166dbdf461d6bc79fd4774"} Oct 31 01:17:48.942129 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:48.941977 4440 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfb5f81ce726830228a41726d33044b98cf710fc0c166dbdf461d6bc79fd4774" Oct 31 01:17:55.995491 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:55.995457 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mxkq4"] Oct 31 01:17:55.996001 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:55.995894 4440 cpu_manager.go:414] "containerMap: removing container" podUID="33be65c2-e4d4-4fd8-bf80-1c3d9be57bee" containerName="revoke-keys" Oct 31 01:17:55.996001 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:55.995908 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33be65c2-e4d4-4fd8-bf80-1c3d9be57bee" containerName="revoke-keys" Oct 31 01:17:55.996180 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:55.996016 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="33be65c2-e4d4-4fd8-bf80-1c3d9be57bee" containerName="revoke-keys" Oct 31 01:17:55.999780 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:55.999753 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mxkq4" Oct 31 01:17:56.013474 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:56.013446 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:17:56.013610 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:56.013527 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:17:56.013610 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:56.013601 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:17:56.013955 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:56.013926 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zs6l6\"" Oct 31 01:17:56.017964 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:56.017936 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mxkq4"] Oct 31 01:17:56.167875 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:56.167830 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpsbm\" (UniqueName: \"kubernetes.io/projected/7e76c45a-690e-4796-ae06-6c5147d9990e-kube-api-access-lpsbm\") pod \"maas-api-revoke-sub-d3fba056-mxkq4\" (UID: \"7e76c45a-690e-4796-ae06-6c5147d9990e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mxkq4" Oct 31 01:17:56.168061 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:56.167957 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e76c45a-690e-4796-ae06-6c5147d9990e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-mxkq4\" (UID: \"7e76c45a-690e-4796-ae06-6c5147d9990e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mxkq4" Oct 31 01:17:56.168061 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:56.167997 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e76c45a-690e-4796-ae06-6c5147d9990e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-mxkq4\" (UID: \"7e76c45a-690e-4796-ae06-6c5147d9990e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mxkq4" Oct 31 01:17:56.269811 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:56.269716 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpsbm\" (UniqueName: \"kubernetes.io/projected/7e76c45a-690e-4796-ae06-6c5147d9990e-kube-api-access-lpsbm\") pod \"maas-api-revoke-sub-d3fba056-mxkq4\" (UID: \"7e76c45a-690e-4796-ae06-6c5147d9990e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mxkq4" Oct 31 01:17:56.269995 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:56.269834 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e76c45a-690e-4796-ae06-6c5147d9990e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-mxkq4\" (UID: \"7e76c45a-690e-4796-ae06-6c5147d9990e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mxkq4" Oct 31 01:17:56.269995 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:56.269878 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e76c45a-690e-4796-ae06-6c5147d9990e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-mxkq4\" (UID: \"7e76c45a-690e-4796-ae06-6c5147d9990e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mxkq4" Oct 31 01:17:56.271238 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:56.271209 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e76c45a-690e-4796-ae06-6c5147d9990e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-mxkq4\" (UID: \"7e76c45a-690e-4796-ae06-6c5147d9990e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mxkq4" Oct 31 01:17:56.273330 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:56.273306 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e76c45a-690e-4796-ae06-6c5147d9990e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-mxkq4\" (UID: \"7e76c45a-690e-4796-ae06-6c5147d9990e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mxkq4" Oct 31 01:17:56.277478 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:56.277455 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpsbm\" (UniqueName: \"kubernetes.io/projected/7e76c45a-690e-4796-ae06-6c5147d9990e-kube-api-access-lpsbm\") pod \"maas-api-revoke-sub-d3fba056-mxkq4\" (UID: \"7e76c45a-690e-4796-ae06-6c5147d9990e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mxkq4" Oct 31 01:17:56.315310 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:56.315278 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mxkq4" Oct 31 01:17:56.564520 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:56.564444 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:17:56.594379 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:56.594348 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mxkq4"] Oct 31 01:17:56.595157 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:17:56.595130 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e76c45a_690e_4796_ae06_6c5147d9990e.slice/crio-bb9c86012f36a37c5f6cbc48a8b06e1f76f5805209afe9f7b65600fdc7456314 WatchSource:0}: Error finding container bb9c86012f36a37c5f6cbc48a8b06e1f76f5805209afe9f7b65600fdc7456314: Status 404 returned error can't find the container with id bb9c86012f36a37c5f6cbc48a8b06e1f76f5805209afe9f7b65600fdc7456314 Oct 31 01:17:56.994777 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:56.994732 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mxkq4" event={"ID":"7e76c45a-690e-4796-ae06-6c5147d9990e","Type":"ContainerStarted","Data":"15c996cea89ee36d41cfcf1d71c01a2487086e21060b48686650033646cc140f"} Oct 31 01:17:56.994995 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:56.994815 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mxkq4" event={"ID":"7e76c45a-690e-4796-ae06-6c5147d9990e","Type":"ContainerStarted","Data":"bb9c86012f36a37c5f6cbc48a8b06e1f76f5805209afe9f7b65600fdc7456314"} Oct 31 01:17:57.013090 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:57.013016 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mxkq4" podStartSLOduration=2.012994626 podStartE2EDuration="2.012994626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:17:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:17:57.010505078 +0000 UTC m=+1321.369227394" watchObservedRunningTime="2026-10-31 01:17:57.012994626 +0000 UTC m=+1321.371716942" Oct 31 01:17:58.022439 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:58.022370 4440 generic.go:356] "Generic (PLEG): container finished" podID="7e76c45a-690e-4796-ae06-6c5147d9990e" containerID="15c996cea89ee36d41cfcf1d71c01a2487086e21060b48686650033646cc140f" exitCode=0 Oct 31 01:17:58.023598 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:58.023405 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mxkq4" event={"ID":"7e76c45a-690e-4796-ae06-6c5147d9990e","Type":"ContainerDied","Data":"15c996cea89ee36d41cfcf1d71c01a2487086e21060b48686650033646cc140f"} Oct 31 01:17:59.974361 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:17:59.974328 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mxkq4" Oct 31 01:18:00.059493 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:00.059087 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mxkq4" event={"ID":"7e76c45a-690e-4796-ae06-6c5147d9990e","Type":"ContainerDied","Data":"bb9c86012f36a37c5f6cbc48a8b06e1f76f5805209afe9f7b65600fdc7456314"} Oct 31 01:18:00.059493 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:00.059158 4440 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb9c86012f36a37c5f6cbc48a8b06e1f76f5805209afe9f7b65600fdc7456314" Oct 31 01:18:00.059493 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:00.059237 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mxkq4" Oct 31 01:18:00.115076 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:00.115023 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e76c45a-690e-4796-ae06-6c5147d9990e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e76c45a-690e-4796-ae06-6c5147d9990e-maas-api-openshift-ca\") pod \"7e76c45a-690e-4796-ae06-6c5147d9990e\" (UID: \"7e76c45a-690e-4796-ae06-6c5147d9990e\") " Oct 31 01:18:00.115301 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:00.115182 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e76c45a-690e-4796-ae06-6c5147d9990e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e76c45a-690e-4796-ae06-6c5147d9990e-maas-api-xks-ca\") pod \"7e76c45a-690e-4796-ae06-6c5147d9990e\" (UID: \"7e76c45a-690e-4796-ae06-6c5147d9990e\") " Oct 31 01:18:00.115301 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:00.115230 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e76c45a-690e-4796-ae06-6c5147d9990e-kube-api-access-lpsbm\" (UniqueName: \"kubernetes.io/projected/7e76c45a-690e-4796-ae06-6c5147d9990e-kube-api-access-lpsbm\") pod \"7e76c45a-690e-4796-ae06-6c5147d9990e\" (UID: \"7e76c45a-690e-4796-ae06-6c5147d9990e\") " Oct 31 01:18:00.115955 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:00.115924 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e76c45a-690e-4796-ae06-6c5147d9990e-maas-api-openshift-ca" pod "7e76c45a-690e-4796-ae06-6c5147d9990e" (UID: "7e76c45a-690e-4796-ae06-6c5147d9990e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:18:00.119266 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:00.119176 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e76c45a-690e-4796-ae06-6c5147d9990e-maas-api-xks-ca" pod "7e76c45a-690e-4796-ae06-6c5147d9990e" (UID: "7e76c45a-690e-4796-ae06-6c5147d9990e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:18:00.120306 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:00.120271 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e76c45a-690e-4796-ae06-6c5147d9990e-kube-api-access-lpsbm" pod "7e76c45a-690e-4796-ae06-6c5147d9990e" (UID: "7e76c45a-690e-4796-ae06-6c5147d9990e"). InnerVolumeSpecName "kube-api-access-lpsbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:18:00.217192 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:00.217149 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpsbm\" (UniqueName: \"kubernetes.io/projected/7e76c45a-690e-4796-ae06-6c5147d9990e-kube-api-access-lpsbm\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:18:00.217192 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:00.217187 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e76c45a-690e-4796-ae06-6c5147d9990e-maas-api-openshift-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:18:00.217192 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:00.217198 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e76c45a-690e-4796-ae06-6c5147d9990e-maas-api-xks-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:18:01.003243 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:18:01.003197 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:18:06.848734 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:06.848694 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:18:11.146442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:11.146059 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:18:16.706705 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:16.706665 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xqs8b"] Oct 31 01:18:16.707179 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:16.707165 4440 cpu_manager.go:414] "containerMap: removing container" podUID="7e76c45a-690e-4796-ae06-6c5147d9990e" containerName="revoke-keys" Oct 31 01:18:16.707244 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:16.707180 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e76c45a-690e-4796-ae06-6c5147d9990e" containerName="revoke-keys" Oct 31 01:18:16.707280 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:16.707253 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e76c45a-690e-4796-ae06-6c5147d9990e" containerName="revoke-keys" Oct 31 01:18:16.710573 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:16.710552 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xqs8b" Oct 31 01:18:16.715387 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:16.715288 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:18:16.715596 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:16.715572 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:18:16.715805 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:16.715654 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zs6l6\"" Oct 31 01:18:16.715805 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:16.715726 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:18:16.720731 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:16.720704 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xqs8b"] Oct 31 01:18:16.839951 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:16.839917 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r88sl\" (UniqueName: \"kubernetes.io/projected/4d32e347-1610-48e9-bcbb-25005fc2e92a-kube-api-access-r88sl\") pod \"maas-api-revoke-sub-ae594979-xqs8b\" (UID: \"4d32e347-1610-48e9-bcbb-25005fc2e92a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xqs8b" Oct 31 01:18:16.840144 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:16.839975 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d32e347-1610-48e9-bcbb-25005fc2e92a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-xqs8b\" (UID: \"4d32e347-1610-48e9-bcbb-25005fc2e92a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xqs8b" Oct 31 01:18:16.840144 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:16.840088 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d32e347-1610-48e9-bcbb-25005fc2e92a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-xqs8b\" (UID: \"4d32e347-1610-48e9-bcbb-25005fc2e92a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xqs8b" Oct 31 01:18:16.940899 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:16.940860 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d32e347-1610-48e9-bcbb-25005fc2e92a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-xqs8b\" (UID: \"4d32e347-1610-48e9-bcbb-25005fc2e92a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xqs8b" Oct 31 01:18:16.941097 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:16.940940 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r88sl\" (UniqueName: \"kubernetes.io/projected/4d32e347-1610-48e9-bcbb-25005fc2e92a-kube-api-access-r88sl\") pod \"maas-api-revoke-sub-ae594979-xqs8b\" (UID: \"4d32e347-1610-48e9-bcbb-25005fc2e92a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xqs8b" Oct 31 01:18:16.941097 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:16.940970 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d32e347-1610-48e9-bcbb-25005fc2e92a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-xqs8b\" (UID: \"4d32e347-1610-48e9-bcbb-25005fc2e92a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xqs8b" Oct 31 01:18:16.941552 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:16.941527 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d32e347-1610-48e9-bcbb-25005fc2e92a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-xqs8b\" (UID: \"4d32e347-1610-48e9-bcbb-25005fc2e92a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xqs8b" Oct 31 01:18:16.943354 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:16.943321 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d32e347-1610-48e9-bcbb-25005fc2e92a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-xqs8b\" (UID: \"4d32e347-1610-48e9-bcbb-25005fc2e92a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xqs8b" Oct 31 01:18:16.948266 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:16.948240 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r88sl\" (UniqueName: \"kubernetes.io/projected/4d32e347-1610-48e9-bcbb-25005fc2e92a-kube-api-access-r88sl\") pod \"maas-api-revoke-sub-ae594979-xqs8b\" (UID: \"4d32e347-1610-48e9-bcbb-25005fc2e92a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xqs8b" Oct 31 01:18:17.029261 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:17.029167 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xqs8b" Oct 31 01:18:17.308085 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:17.308054 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xqs8b"] Oct 31 01:18:17.309350 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:18:17.309322 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d32e347_1610_48e9_bcbb_25005fc2e92a.slice/crio-b4996988a5a33b12168f24467848f408159f5e24dc0f89ceb7a2106d61d10347 WatchSource:0}: Error finding container b4996988a5a33b12168f24467848f408159f5e24dc0f89ceb7a2106d61d10347: Status 404 returned error can't find the container with id b4996988a5a33b12168f24467848f408159f5e24dc0f89ceb7a2106d61d10347 Oct 31 01:18:18.235583 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:18.235542 4440 generic.go:356] "Generic (PLEG): container finished" podID="4d32e347-1610-48e9-bcbb-25005fc2e92a" containerID="9ebe7641ec200120fd7eb5586317aafd63776719b1af84b52aed61459691b1af" exitCode=0 Oct 31 01:18:18.235975 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:18.235666 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xqs8b" event={"ID":"4d32e347-1610-48e9-bcbb-25005fc2e92a","Type":"ContainerDied","Data":"9ebe7641ec200120fd7eb5586317aafd63776719b1af84b52aed61459691b1af"} Oct 31 01:18:18.235975 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:18.235725 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xqs8b" event={"ID":"4d32e347-1610-48e9-bcbb-25005fc2e92a","Type":"ContainerStarted","Data":"b4996988a5a33b12168f24467848f408159f5e24dc0f89ceb7a2106d61d10347"} Oct 31 01:18:19.380110 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:19.380079 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xqs8b" Oct 31 01:18:19.567362 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:19.567258 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d32e347-1610-48e9-bcbb-25005fc2e92a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d32e347-1610-48e9-bcbb-25005fc2e92a-maas-api-xks-ca\") pod \"4d32e347-1610-48e9-bcbb-25005fc2e92a\" (UID: \"4d32e347-1610-48e9-bcbb-25005fc2e92a\") " Oct 31 01:18:19.567362 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:19.567355 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d32e347-1610-48e9-bcbb-25005fc2e92a-kube-api-access-r88sl\" (UniqueName: \"kubernetes.io/projected/4d32e347-1610-48e9-bcbb-25005fc2e92a-kube-api-access-r88sl\") pod \"4d32e347-1610-48e9-bcbb-25005fc2e92a\" (UID: \"4d32e347-1610-48e9-bcbb-25005fc2e92a\") " Oct 31 01:18:19.567624 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:19.567487 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d32e347-1610-48e9-bcbb-25005fc2e92a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d32e347-1610-48e9-bcbb-25005fc2e92a-maas-api-openshift-ca\") pod \"4d32e347-1610-48e9-bcbb-25005fc2e92a\" (UID: \"4d32e347-1610-48e9-bcbb-25005fc2e92a\") " Oct 31 01:18:19.567903 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:19.567869 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d32e347-1610-48e9-bcbb-25005fc2e92a-maas-api-openshift-ca" pod "4d32e347-1610-48e9-bcbb-25005fc2e92a" (UID: "4d32e347-1610-48e9-bcbb-25005fc2e92a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:18:19.569519 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:19.569495 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d32e347-1610-48e9-bcbb-25005fc2e92a-maas-api-xks-ca" pod "4d32e347-1610-48e9-bcbb-25005fc2e92a" (UID: "4d32e347-1610-48e9-bcbb-25005fc2e92a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:18:19.569595 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:19.569526 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d32e347-1610-48e9-bcbb-25005fc2e92a-kube-api-access-r88sl" pod "4d32e347-1610-48e9-bcbb-25005fc2e92a" (UID: "4d32e347-1610-48e9-bcbb-25005fc2e92a"). InnerVolumeSpecName "kube-api-access-r88sl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:18:19.668471 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:19.668402 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d32e347-1610-48e9-bcbb-25005fc2e92a-maas-api-xks-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:18:19.668471 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:19.668466 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r88sl\" (UniqueName: \"kubernetes.io/projected/4d32e347-1610-48e9-bcbb-25005fc2e92a-kube-api-access-r88sl\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:18:19.668471 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:19.668477 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d32e347-1610-48e9-bcbb-25005fc2e92a-maas-api-openshift-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:18:20.249370 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:20.249340 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xqs8b" Oct 31 01:18:20.249563 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:20.249340 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xqs8b" event={"ID":"4d32e347-1610-48e9-bcbb-25005fc2e92a","Type":"ContainerDied","Data":"b4996988a5a33b12168f24467848f408159f5e24dc0f89ceb7a2106d61d10347"} Oct 31 01:18:20.249563 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:20.249466 4440 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4996988a5a33b12168f24467848f408159f5e24dc0f89ceb7a2106d61d10347" Oct 31 01:18:27.572089 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:27.572039 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:18:31.110029 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:18:31.109982 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:18:31.716458 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:31.716395 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:18:34.791255 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:34.791217 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5kf2p"] Oct 31 01:18:34.791694 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:34.791680 4440 cpu_manager.go:414] "containerMap: removing container" podUID="4d32e347-1610-48e9-bcbb-25005fc2e92a" containerName="revoke-keys" Oct 31 01:18:34.791745 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:34.791696 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d32e347-1610-48e9-bcbb-25005fc2e92a" containerName="revoke-keys" Oct 31 01:18:34.791795 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:34.791786 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d32e347-1610-48e9-bcbb-25005fc2e92a" containerName="revoke-keys" Oct 31 01:18:34.796324 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:34.796302 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5kf2p" Oct 31 01:18:34.802111 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:34.802074 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5kf2p"] Oct 31 01:18:34.802590 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:34.802565 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zs6l6\"" Oct 31 01:18:34.802726 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:34.802589 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:18:34.830136 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:34.830102 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:18:34.830136 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:34.830128 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:18:34.890889 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:34.890841 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae2356b6-2a2e-48b2-b1a4-4856586c3e36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-5kf2p\" (UID: \"ae2356b6-2a2e-48b2-b1a4-4856586c3e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5kf2p" Oct 31 01:18:34.891103 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:34.890907 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae2356b6-2a2e-48b2-b1a4-4856586c3e36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-5kf2p\" (UID: \"ae2356b6-2a2e-48b2-b1a4-4856586c3e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5kf2p" Oct 31 01:18:34.891103 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:34.891078 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgw2l\" (UniqueName: \"kubernetes.io/projected/ae2356b6-2a2e-48b2-b1a4-4856586c3e36-kube-api-access-qgw2l\") pod \"maas-api-revoke-sub-a4aef7d7-5kf2p\" (UID: \"ae2356b6-2a2e-48b2-b1a4-4856586c3e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5kf2p" Oct 31 01:18:34.992167 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:34.992127 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgw2l\" (UniqueName: \"kubernetes.io/projected/ae2356b6-2a2e-48b2-b1a4-4856586c3e36-kube-api-access-qgw2l\") pod \"maas-api-revoke-sub-a4aef7d7-5kf2p\" (UID: \"ae2356b6-2a2e-48b2-b1a4-4856586c3e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5kf2p" Oct 31 01:18:34.992378 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:34.992226 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae2356b6-2a2e-48b2-b1a4-4856586c3e36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-5kf2p\" (UID: \"ae2356b6-2a2e-48b2-b1a4-4856586c3e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5kf2p" Oct 31 01:18:34.992378 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:34.992268 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae2356b6-2a2e-48b2-b1a4-4856586c3e36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-5kf2p\" (UID: \"ae2356b6-2a2e-48b2-b1a4-4856586c3e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5kf2p" Oct 31 01:18:34.992936 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:34.992915 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae2356b6-2a2e-48b2-b1a4-4856586c3e36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-5kf2p\" (UID: \"ae2356b6-2a2e-48b2-b1a4-4856586c3e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5kf2p" Oct 31 01:18:34.994690 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:34.994663 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae2356b6-2a2e-48b2-b1a4-4856586c3e36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-5kf2p\" (UID: \"ae2356b6-2a2e-48b2-b1a4-4856586c3e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5kf2p" Oct 31 01:18:34.999932 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:34.999907 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgw2l\" (UniqueName: \"kubernetes.io/projected/ae2356b6-2a2e-48b2-b1a4-4856586c3e36-kube-api-access-qgw2l\") pod \"maas-api-revoke-sub-a4aef7d7-5kf2p\" (UID: \"ae2356b6-2a2e-48b2-b1a4-4856586c3e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5kf2p" Oct 31 01:18:35.117410 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:35.117332 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5kf2p" Oct 31 01:18:35.394467 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:35.394439 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5kf2p"] Oct 31 01:18:35.395581 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:18:35.395542 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae2356b6_2a2e_48b2_b1a4_4856586c3e36.slice/crio-3b8eea55406b0f38db92c719f75ec032880555dff76391cb8b5d153cd77c85dd WatchSource:0}: Error finding container 3b8eea55406b0f38db92c719f75ec032880555dff76391cb8b5d153cd77c85dd: Status 404 returned error can't find the container with id 3b8eea55406b0f38db92c719f75ec032880555dff76391cb8b5d153cd77c85dd Oct 31 01:18:36.384739 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:36.384697 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5kf2p" event={"ID":"ae2356b6-2a2e-48b2-b1a4-4856586c3e36","Type":"ContainerStarted","Data":"8aa297936514083962b0472441ea159c4a27707240c45e2f5d323900f1b95e9e"} Oct 31 01:18:36.385162 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:36.384793 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5kf2p" event={"ID":"ae2356b6-2a2e-48b2-b1a4-4856586c3e36","Type":"ContainerStarted","Data":"3b8eea55406b0f38db92c719f75ec032880555dff76391cb8b5d153cd77c85dd"} Oct 31 01:18:36.407994 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:36.407929 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5kf2p" podStartSLOduration=2.407904178 podStartE2EDuration="2.407904178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:18:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:18:36.401359981 +0000 UTC m=+1360.760082322" watchObservedRunningTime="2026-10-31 01:18:36.407904178 +0000 UTC m=+1360.766626497" Oct 31 01:18:37.392943 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:37.392902 4440 generic.go:356] "Generic (PLEG): container finished" podID="ae2356b6-2a2e-48b2-b1a4-4856586c3e36" containerID="8aa297936514083962b0472441ea159c4a27707240c45e2f5d323900f1b95e9e" exitCode=0 Oct 31 01:18:37.393381 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:37.392980 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5kf2p" event={"ID":"ae2356b6-2a2e-48b2-b1a4-4856586c3e36","Type":"ContainerDied","Data":"8aa297936514083962b0472441ea159c4a27707240c45e2f5d323900f1b95e9e"} Oct 31 01:18:38.537016 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:38.536989 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5kf2p" Oct 31 01:18:38.630584 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:38.630538 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae2356b6-2a2e-48b2-b1a4-4856586c3e36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae2356b6-2a2e-48b2-b1a4-4856586c3e36-maas-api-openshift-ca\") pod \"ae2356b6-2a2e-48b2-b1a4-4856586c3e36\" (UID: \"ae2356b6-2a2e-48b2-b1a4-4856586c3e36\") " Oct 31 01:18:38.630584 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:38.630600 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae2356b6-2a2e-48b2-b1a4-4856586c3e36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae2356b6-2a2e-48b2-b1a4-4856586c3e36-maas-api-xks-ca\") pod \"ae2356b6-2a2e-48b2-b1a4-4856586c3e36\" (UID: \"ae2356b6-2a2e-48b2-b1a4-4856586c3e36\") " Oct 31 01:18:38.630800 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:38.630646 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae2356b6-2a2e-48b2-b1a4-4856586c3e36-kube-api-access-qgw2l\" (UniqueName: \"kubernetes.io/projected/ae2356b6-2a2e-48b2-b1a4-4856586c3e36-kube-api-access-qgw2l\") pod \"ae2356b6-2a2e-48b2-b1a4-4856586c3e36\" (UID: \"ae2356b6-2a2e-48b2-b1a4-4856586c3e36\") " Oct 31 01:18:38.630981 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:38.630955 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae2356b6-2a2e-48b2-b1a4-4856586c3e36-maas-api-openshift-ca" pod "ae2356b6-2a2e-48b2-b1a4-4856586c3e36" (UID: "ae2356b6-2a2e-48b2-b1a4-4856586c3e36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:18:38.632806 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:38.632773 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae2356b6-2a2e-48b2-b1a4-4856586c3e36-maas-api-xks-ca" pod "ae2356b6-2a2e-48b2-b1a4-4856586c3e36" (UID: "ae2356b6-2a2e-48b2-b1a4-4856586c3e36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:18:38.632933 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:38.632850 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae2356b6-2a2e-48b2-b1a4-4856586c3e36-kube-api-access-qgw2l" pod "ae2356b6-2a2e-48b2-b1a4-4856586c3e36" (UID: "ae2356b6-2a2e-48b2-b1a4-4856586c3e36"). InnerVolumeSpecName "kube-api-access-qgw2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:18:38.732346 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:38.732316 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgw2l\" (UniqueName: \"kubernetes.io/projected/ae2356b6-2a2e-48b2-b1a4-4856586c3e36-kube-api-access-qgw2l\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:18:38.732346 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:38.732343 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae2356b6-2a2e-48b2-b1a4-4856586c3e36-maas-api-openshift-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:18:38.732563 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:38.732354 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae2356b6-2a2e-48b2-b1a4-4856586c3e36-maas-api-xks-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:18:39.406890 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:39.406859 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5kf2p" Oct 31 01:18:39.407082 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:39.406857 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5kf2p" event={"ID":"ae2356b6-2a2e-48b2-b1a4-4856586c3e36","Type":"ContainerDied","Data":"3b8eea55406b0f38db92c719f75ec032880555dff76391cb8b5d153cd77c85dd"} Oct 31 01:18:39.407082 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:39.406983 4440 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b8eea55406b0f38db92c719f75ec032880555dff76391cb8b5d153cd77c85dd" Oct 31 01:18:45.606075 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:45.606032 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:18:46.926985 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:46.926948 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:47.116824 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:47.116782 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:47.234138 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:47.234041 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:51.356362 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:51.356310 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:18:53.530107 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:53.530064 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hbhgd"] Oct 31 01:18:53.530764 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:53.530741 4440 cpu_manager.go:414] "containerMap: removing container" podUID="ae2356b6-2a2e-48b2-b1a4-4856586c3e36" containerName="revoke-keys" Oct 31 01:18:53.530825 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:53.530770 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae2356b6-2a2e-48b2-b1a4-4856586c3e36" containerName="revoke-keys" Oct 31 01:18:53.530920 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:53.530908 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae2356b6-2a2e-48b2-b1a4-4856586c3e36" containerName="revoke-keys" Oct 31 01:18:53.534766 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:53.534743 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hbhgd" Oct 31 01:18:53.538923 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:53.538734 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:18:53.538923 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:53.538808 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:18:53.538923 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:53.538897 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zs6l6\"" Oct 31 01:18:53.539882 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:53.539851 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:18:53.540293 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:53.540264 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hbhgd"] Oct 31 01:18:53.627816 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:53.627773 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33492813-313f-430e-bf1f-6676b16e42b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-hbhgd\" (UID: \"33492813-313f-430e-bf1f-6676b16e42b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hbhgd" Oct 31 01:18:53.628001 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:53.627948 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33492813-313f-430e-bf1f-6676b16e42b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-hbhgd\" (UID: \"33492813-313f-430e-bf1f-6676b16e42b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hbhgd" Oct 31 01:18:53.628075 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:53.628054 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s28wb\" (UniqueName: \"kubernetes.io/projected/33492813-313f-430e-bf1f-6676b16e42b8-kube-api-access-s28wb\") pod \"maas-api-revoke-sub-79805ca9-hbhgd\" (UID: \"33492813-313f-430e-bf1f-6676b16e42b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hbhgd" Oct 31 01:18:53.729133 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:53.729081 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s28wb\" (UniqueName: \"kubernetes.io/projected/33492813-313f-430e-bf1f-6676b16e42b8-kube-api-access-s28wb\") pod \"maas-api-revoke-sub-79805ca9-hbhgd\" (UID: \"33492813-313f-430e-bf1f-6676b16e42b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hbhgd" Oct 31 01:18:53.729361 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:53.729162 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33492813-313f-430e-bf1f-6676b16e42b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-hbhgd\" (UID: \"33492813-313f-430e-bf1f-6676b16e42b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hbhgd" Oct 31 01:18:53.729361 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:53.729292 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33492813-313f-430e-bf1f-6676b16e42b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-hbhgd\" (UID: \"33492813-313f-430e-bf1f-6676b16e42b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hbhgd" Oct 31 01:18:53.729915 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:53.729885 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33492813-313f-430e-bf1f-6676b16e42b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-hbhgd\" (UID: \"33492813-313f-430e-bf1f-6676b16e42b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hbhgd" Oct 31 01:18:53.731715 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:53.731689 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33492813-313f-430e-bf1f-6676b16e42b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-hbhgd\" (UID: \"33492813-313f-430e-bf1f-6676b16e42b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hbhgd" Oct 31 01:18:53.737524 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:53.737494 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s28wb\" (UniqueName: \"kubernetes.io/projected/33492813-313f-430e-bf1f-6676b16e42b8-kube-api-access-s28wb\") pod \"maas-api-revoke-sub-79805ca9-hbhgd\" (UID: \"33492813-313f-430e-bf1f-6676b16e42b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hbhgd" Oct 31 01:18:53.853620 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:53.853529 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hbhgd" Oct 31 01:18:54.131945 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:54.131911 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hbhgd"] Oct 31 01:18:54.132825 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:18:54.132790 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33492813_313f_430e_bf1f_6676b16e42b8.slice/crio-1e50ebececb84201ef304e569c81eb72631be8bc58aaeddcc1d63e2d1ef09ad1 WatchSource:0}: Error finding container 1e50ebececb84201ef304e569c81eb72631be8bc58aaeddcc1d63e2d1ef09ad1: Status 404 returned error can't find the container with id 1e50ebececb84201ef304e569c81eb72631be8bc58aaeddcc1d63e2d1ef09ad1 Oct 31 01:18:54.557902 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:54.557860 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hbhgd" event={"ID":"33492813-313f-430e-bf1f-6676b16e42b8","Type":"ContainerStarted","Data":"9b7dc877756bd44ef405821e4358cf77d1150c395003fdb2c105e2f4b0ee4e0c"} Oct 31 01:18:54.558344 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:54.557924 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hbhgd" event={"ID":"33492813-313f-430e-bf1f-6676b16e42b8","Type":"ContainerStarted","Data":"1e50ebececb84201ef304e569c81eb72631be8bc58aaeddcc1d63e2d1ef09ad1"} Oct 31 01:18:54.575820 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:54.575771 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hbhgd" podStartSLOduration=1.575757638 podStartE2EDuration="1.575757638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:18:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:18:54.572981915 +0000 UTC m=+1378.931704234" watchObservedRunningTime="2026-10-31 01:18:54.575757638 +0000 UTC m=+1378.934479954" Oct 31 01:18:55.566137 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:55.566102 4440 generic.go:356] "Generic (PLEG): container finished" podID="33492813-313f-430e-bf1f-6676b16e42b8" containerID="9b7dc877756bd44ef405821e4358cf77d1150c395003fdb2c105e2f4b0ee4e0c" exitCode=0 Oct 31 01:18:55.566577 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:55.566170 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hbhgd" event={"ID":"33492813-313f-430e-bf1f-6676b16e42b8","Type":"ContainerDied","Data":"9b7dc877756bd44ef405821e4358cf77d1150c395003fdb2c105e2f4b0ee4e0c"} Oct 31 01:18:56.558907 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:56.558821 4440 scope.go:122] "RemoveContainer" containerID="48f30cd271402c703cf948c7a6175c8b5b2778e52a098ec52cae25c96ee06d4b" Oct 31 01:18:56.708330 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:56.708303 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hbhgd" Oct 31 01:18:56.860249 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:56.860162 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33492813-313f-430e-bf1f-6676b16e42b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33492813-313f-430e-bf1f-6676b16e42b8-maas-api-openshift-ca\") pod \"33492813-313f-430e-bf1f-6676b16e42b8\" (UID: \"33492813-313f-430e-bf1f-6676b16e42b8\") " Oct 31 01:18:56.860249 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:56.860202 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33492813-313f-430e-bf1f-6676b16e42b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33492813-313f-430e-bf1f-6676b16e42b8-maas-api-xks-ca\") pod \"33492813-313f-430e-bf1f-6676b16e42b8\" (UID: \"33492813-313f-430e-bf1f-6676b16e42b8\") " Oct 31 01:18:56.860506 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:56.860286 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33492813-313f-430e-bf1f-6676b16e42b8-kube-api-access-s28wb\" (UniqueName: \"kubernetes.io/projected/33492813-313f-430e-bf1f-6676b16e42b8-kube-api-access-s28wb\") pod \"33492813-313f-430e-bf1f-6676b16e42b8\" (UID: \"33492813-313f-430e-bf1f-6676b16e42b8\") " Oct 31 01:18:56.860643 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:56.860615 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33492813-313f-430e-bf1f-6676b16e42b8-maas-api-openshift-ca" pod "33492813-313f-430e-bf1f-6676b16e42b8" (UID: "33492813-313f-430e-bf1f-6676b16e42b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:18:56.862461 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:56.862412 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33492813-313f-430e-bf1f-6676b16e42b8-maas-api-xks-ca" pod "33492813-313f-430e-bf1f-6676b16e42b8" (UID: "33492813-313f-430e-bf1f-6676b16e42b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:18:56.862461 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:56.862438 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33492813-313f-430e-bf1f-6676b16e42b8-kube-api-access-s28wb" pod "33492813-313f-430e-bf1f-6676b16e42b8" (UID: "33492813-313f-430e-bf1f-6676b16e42b8"). InnerVolumeSpecName "kube-api-access-s28wb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:18:56.961188 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:56.961163 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s28wb\" (UniqueName: \"kubernetes.io/projected/33492813-313f-430e-bf1f-6676b16e42b8-kube-api-access-s28wb\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:18:56.961188 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:56.961186 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33492813-313f-430e-bf1f-6676b16e42b8-maas-api-openshift-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:18:56.961341 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:56.961196 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33492813-313f-430e-bf1f-6676b16e42b8-maas-api-xks-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:18:57.580047 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:57.580006 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hbhgd" Oct 31 01:18:57.580047 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:57.580030 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hbhgd" event={"ID":"33492813-313f-430e-bf1f-6676b16e42b8","Type":"ContainerDied","Data":"1e50ebececb84201ef304e569c81eb72631be8bc58aaeddcc1d63e2d1ef09ad1"} Oct 31 01:18:57.580317 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:57.580072 4440 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e50ebececb84201ef304e569c81eb72631be8bc58aaeddcc1d63e2d1ef09ad1" Oct 31 01:18:58.022603 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:58.022548 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10c0b05c-d7ntq"] Oct 31 01:18:58.027845 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:58.027816 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10c0b05c-d7ntq"] Oct 31 01:18:59.769998 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:18:59.769965 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8634e16-a396-4242-9efb-0c398713cb77" path="/var/lib/kubelet/pods/c8634e16-a396-4242-9efb-0c398713cb77/volumes" Oct 31 01:19:00.999539 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:19:00.999503 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:19:04.388848 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:04.388815 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:19:08.572460 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:08.568777 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:19:11.301386 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:11.301333 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k84wf"] Oct 31 01:19:11.302062 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:11.302039 4440 cpu_manager.go:414] "containerMap: removing container" podUID="33492813-313f-430e-bf1f-6676b16e42b8" containerName="revoke-keys" Oct 31 01:19:11.302111 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:11.302067 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33492813-313f-430e-bf1f-6676b16e42b8" containerName="revoke-keys" Oct 31 01:19:11.302212 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:11.302199 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="33492813-313f-430e-bf1f-6676b16e42b8" containerName="revoke-keys" Oct 31 01:19:11.307623 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:11.307437 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k84wf" Oct 31 01:19:11.312824 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:11.312773 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k84wf"] Oct 31 01:19:11.314226 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:11.314181 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:19:11.314705 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:11.314604 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zs6l6\"" Oct 31 01:19:11.314705 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:11.314656 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:19:11.314705 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:11.314668 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:19:11.356844 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:11.356801 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc5d14b8-cd36-4d76-a8f7-4e000c6238ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-k84wf\" (UID: \"fc5d14b8-cd36-4d76-a8f7-4e000c6238ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k84wf" Oct 31 01:19:11.357064 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:11.356873 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk77d\" (UniqueName: \"kubernetes.io/projected/fc5d14b8-cd36-4d76-a8f7-4e000c6238ec-kube-api-access-qk77d\") pod \"maas-api-revoke-sub-0b7e8edb-k84wf\" (UID: \"fc5d14b8-cd36-4d76-a8f7-4e000c6238ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k84wf" Oct 31 01:19:11.357064 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:11.356998 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc5d14b8-cd36-4d76-a8f7-4e000c6238ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-k84wf\" (UID: \"fc5d14b8-cd36-4d76-a8f7-4e000c6238ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k84wf" Oct 31 01:19:11.458143 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:11.458104 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc5d14b8-cd36-4d76-a8f7-4e000c6238ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-k84wf\" (UID: \"fc5d14b8-cd36-4d76-a8f7-4e000c6238ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k84wf" Oct 31 01:19:11.458351 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:11.458163 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk77d\" (UniqueName: \"kubernetes.io/projected/fc5d14b8-cd36-4d76-a8f7-4e000c6238ec-kube-api-access-qk77d\") pod \"maas-api-revoke-sub-0b7e8edb-k84wf\" (UID: \"fc5d14b8-cd36-4d76-a8f7-4e000c6238ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k84wf" Oct 31 01:19:11.458351 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:11.458209 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc5d14b8-cd36-4d76-a8f7-4e000c6238ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-k84wf\" (UID: \"fc5d14b8-cd36-4d76-a8f7-4e000c6238ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k84wf" Oct 31 01:19:11.459061 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:11.459031 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc5d14b8-cd36-4d76-a8f7-4e000c6238ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-k84wf\" (UID: \"fc5d14b8-cd36-4d76-a8f7-4e000c6238ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k84wf" Oct 31 01:19:11.460758 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:11.460731 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc5d14b8-cd36-4d76-a8f7-4e000c6238ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-k84wf\" (UID: \"fc5d14b8-cd36-4d76-a8f7-4e000c6238ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k84wf" Oct 31 01:19:11.466719 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:11.466684 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk77d\" (UniqueName: \"kubernetes.io/projected/fc5d14b8-cd36-4d76-a8f7-4e000c6238ec-kube-api-access-qk77d\") pod \"maas-api-revoke-sub-0b7e8edb-k84wf\" (UID: \"fc5d14b8-cd36-4d76-a8f7-4e000c6238ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k84wf" Oct 31 01:19:11.629850 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:11.629754 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k84wf" Oct 31 01:19:11.912179 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:11.912142 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k84wf"] Oct 31 01:19:11.912328 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:19:11.912298 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc5d14b8_cd36_4d76_a8f7_4e000c6238ec.slice/crio-83dbebe4d6b0165a8ed8535cda30a681275953bce42f05fcd118ff1bfc34cb88 WatchSource:0}: Error finding container 83dbebe4d6b0165a8ed8535cda30a681275953bce42f05fcd118ff1bfc34cb88: Status 404 returned error can't find the container with id 83dbebe4d6b0165a8ed8535cda30a681275953bce42f05fcd118ff1bfc34cb88 Oct 31 01:19:12.712683 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:12.712645 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k84wf" event={"ID":"fc5d14b8-cd36-4d76-a8f7-4e000c6238ec","Type":"ContainerStarted","Data":"4c1fc1ead1f7bb25b9eef566e813be40847cb1c1087e2c36c27d637eaf3a6eb5"} Oct 31 01:19:12.713091 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:12.712757 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k84wf" event={"ID":"fc5d14b8-cd36-4d76-a8f7-4e000c6238ec","Type":"ContainerStarted","Data":"83dbebe4d6b0165a8ed8535cda30a681275953bce42f05fcd118ff1bfc34cb88"} Oct 31 01:19:12.741681 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:12.741632 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k84wf" podStartSLOduration=1.7416177720000001 podStartE2EDuration="1.741617772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:19:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:12.73930082 +0000 UTC m=+1397.098023137" watchObservedRunningTime="2026-10-31 01:19:12.741617772 +0000 UTC m=+1397.100340087" Oct 31 01:19:13.721163 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:13.721125 4440 generic.go:356] "Generic (PLEG): container finished" podID="fc5d14b8-cd36-4d76-a8f7-4e000c6238ec" containerID="4c1fc1ead1f7bb25b9eef566e813be40847cb1c1087e2c36c27d637eaf3a6eb5" exitCode=0 Oct 31 01:19:13.721626 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:13.721195 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k84wf" event={"ID":"fc5d14b8-cd36-4d76-a8f7-4e000c6238ec","Type":"ContainerDied","Data":"4c1fc1ead1f7bb25b9eef566e813be40847cb1c1087e2c36c27d637eaf3a6eb5"} Oct 31 01:19:14.864857 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:14.864830 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k84wf" Oct 31 01:19:14.998867 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:14.998778 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc5d14b8-cd36-4d76-a8f7-4e000c6238ec-kube-api-access-qk77d\" (UniqueName: \"kubernetes.io/projected/fc5d14b8-cd36-4d76-a8f7-4e000c6238ec-kube-api-access-qk77d\") pod \"fc5d14b8-cd36-4d76-a8f7-4e000c6238ec\" (UID: \"fc5d14b8-cd36-4d76-a8f7-4e000c6238ec\") " Oct 31 01:19:14.998867 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:14.998846 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc5d14b8-cd36-4d76-a8f7-4e000c6238ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc5d14b8-cd36-4d76-a8f7-4e000c6238ec-maas-api-xks-ca\") pod \"fc5d14b8-cd36-4d76-a8f7-4e000c6238ec\" (UID: \"fc5d14b8-cd36-4d76-a8f7-4e000c6238ec\") " Oct 31 01:19:14.999259 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:14.999012 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc5d14b8-cd36-4d76-a8f7-4e000c6238ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc5d14b8-cd36-4d76-a8f7-4e000c6238ec-maas-api-openshift-ca\") pod \"fc5d14b8-cd36-4d76-a8f7-4e000c6238ec\" (UID: \"fc5d14b8-cd36-4d76-a8f7-4e000c6238ec\") " Oct 31 01:19:14.999342 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:14.999320 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc5d14b8-cd36-4d76-a8f7-4e000c6238ec-maas-api-openshift-ca" pod "fc5d14b8-cd36-4d76-a8f7-4e000c6238ec" (UID: "fc5d14b8-cd36-4d76-a8f7-4e000c6238ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:19:15.001009 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:15.000986 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc5d14b8-cd36-4d76-a8f7-4e000c6238ec-maas-api-xks-ca" pod "fc5d14b8-cd36-4d76-a8f7-4e000c6238ec" (UID: "fc5d14b8-cd36-4d76-a8f7-4e000c6238ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:19:15.001098 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:15.001037 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc5d14b8-cd36-4d76-a8f7-4e000c6238ec-kube-api-access-qk77d" pod "fc5d14b8-cd36-4d76-a8f7-4e000c6238ec" (UID: "fc5d14b8-cd36-4d76-a8f7-4e000c6238ec"). InnerVolumeSpecName "kube-api-access-qk77d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:19:15.029532 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:15.029503 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d7a7c8d-hg9b5"] Oct 31 01:19:15.034431 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:15.034399 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d7a7c8d-hg9b5"] Oct 31 01:19:15.100413 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:15.100380 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc5d14b8-cd36-4d76-a8f7-4e000c6238ec-maas-api-openshift-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:15.100413 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:15.100407 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk77d\" (UniqueName: \"kubernetes.io/projected/fc5d14b8-cd36-4d76-a8f7-4e000c6238ec-kube-api-access-qk77d\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:15.100413 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:15.100431 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc5d14b8-cd36-4d76-a8f7-4e000c6238ec-maas-api-xks-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:15.736142 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:15.736114 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k84wf" Oct 31 01:19:15.736300 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:15.736108 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k84wf" event={"ID":"fc5d14b8-cd36-4d76-a8f7-4e000c6238ec","Type":"ContainerDied","Data":"83dbebe4d6b0165a8ed8535cda30a681275953bce42f05fcd118ff1bfc34cb88"} Oct 31 01:19:15.736300 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:15.736240 4440 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83dbebe4d6b0165a8ed8535cda30a681275953bce42f05fcd118ff1bfc34cb88" Oct 31 01:19:15.770452 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:15.770404 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="478a9053-b8a9-4b70-a96d-aa50644aaf53" path="/var/lib/kubelet/pods/478a9053-b8a9-4b70-a96d-aa50644aaf53/volumes" Oct 31 01:19:21.973989 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:21.973935 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lrphq"] Oct 31 01:19:21.974604 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:21.974582 4440 cpu_manager.go:414] "containerMap: removing container" podUID="fc5d14b8-cd36-4d76-a8f7-4e000c6238ec" containerName="revoke-keys" Oct 31 01:19:21.974722 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:21.974606 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc5d14b8-cd36-4d76-a8f7-4e000c6238ec" containerName="revoke-keys" Oct 31 01:19:21.974780 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:21.974756 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc5d14b8-cd36-4d76-a8f7-4e000c6238ec" containerName="revoke-keys" Oct 31 01:19:21.978373 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:21.978346 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lrphq" Oct 31 01:19:21.985479 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:21.983078 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:19:21.985479 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:21.983084 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:19:21.985479 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:21.983138 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:19:21.985479 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:21.983273 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zs6l6\"" Oct 31 01:19:21.985479 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:21.985386 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lrphq"] Oct 31 01:19:22.077240 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:22.077188 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2c3c6d6-0e64-4ead-aeb5-cc56de00389f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lrphq\" (UID: \"f2c3c6d6-0e64-4ead-aeb5-cc56de00389f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lrphq" Oct 31 01:19:22.077240 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:22.077245 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kzwq\" (UniqueName: \"kubernetes.io/projected/f2c3c6d6-0e64-4ead-aeb5-cc56de00389f-kube-api-access-9kzwq\") pod \"maas-api-revoke-sub-3d4156f1-lrphq\" (UID: \"f2c3c6d6-0e64-4ead-aeb5-cc56de00389f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lrphq" Oct 31 01:19:22.077539 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:22.077385 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2c3c6d6-0e64-4ead-aeb5-cc56de00389f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lrphq\" (UID: \"f2c3c6d6-0e64-4ead-aeb5-cc56de00389f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lrphq" Oct 31 01:19:22.178926 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:22.178884 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2c3c6d6-0e64-4ead-aeb5-cc56de00389f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lrphq\" (UID: \"f2c3c6d6-0e64-4ead-aeb5-cc56de00389f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lrphq" Oct 31 01:19:22.179158 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:22.178936 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kzwq\" (UniqueName: \"kubernetes.io/projected/f2c3c6d6-0e64-4ead-aeb5-cc56de00389f-kube-api-access-9kzwq\") pod \"maas-api-revoke-sub-3d4156f1-lrphq\" (UID: \"f2c3c6d6-0e64-4ead-aeb5-cc56de00389f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lrphq" Oct 31 01:19:22.179158 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:22.179003 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2c3c6d6-0e64-4ead-aeb5-cc56de00389f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lrphq\" (UID: \"f2c3c6d6-0e64-4ead-aeb5-cc56de00389f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lrphq" Oct 31 01:19:22.179689 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:22.179663 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2c3c6d6-0e64-4ead-aeb5-cc56de00389f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lrphq\" (UID: \"f2c3c6d6-0e64-4ead-aeb5-cc56de00389f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lrphq" Oct 31 01:19:22.181628 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:22.181594 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2c3c6d6-0e64-4ead-aeb5-cc56de00389f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lrphq\" (UID: \"f2c3c6d6-0e64-4ead-aeb5-cc56de00389f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lrphq" Oct 31 01:19:22.186623 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:22.186590 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kzwq\" (UniqueName: \"kubernetes.io/projected/f2c3c6d6-0e64-4ead-aeb5-cc56de00389f-kube-api-access-9kzwq\") pod \"maas-api-revoke-sub-3d4156f1-lrphq\" (UID: \"f2c3c6d6-0e64-4ead-aeb5-cc56de00389f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lrphq" Oct 31 01:19:22.192862 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:22.192825 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:19:22.296353 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:22.296237 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lrphq" Oct 31 01:19:22.585505 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:22.585462 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lrphq"] Oct 31 01:19:22.586129 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:19:22.586095 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2c3c6d6_0e64_4ead_aeb5_cc56de00389f.slice/crio-0c6e29da0e7cd4f1acf812fe7a20c8c50e49dc294d08098c9ea62dc61221a802 WatchSource:0}: Error finding container 0c6e29da0e7cd4f1acf812fe7a20c8c50e49dc294d08098c9ea62dc61221a802: Status 404 returned error can't find the container with id 0c6e29da0e7cd4f1acf812fe7a20c8c50e49dc294d08098c9ea62dc61221a802 Oct 31 01:19:22.790168 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:22.790124 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lrphq" event={"ID":"f2c3c6d6-0e64-4ead-aeb5-cc56de00389f","Type":"ContainerStarted","Data":"0c6e29da0e7cd4f1acf812fe7a20c8c50e49dc294d08098c9ea62dc61221a802"} Oct 31 01:19:23.798345 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:23.798255 4440 generic.go:356] "Generic (PLEG): container finished" podID="f2c3c6d6-0e64-4ead-aeb5-cc56de00389f" containerID="f4431a0fe49c22f8a9a008a679002d680737f5160c684e9a4783522afdf557dd" exitCode=0 Oct 31 01:19:23.798741 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:23.798337 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lrphq" event={"ID":"f2c3c6d6-0e64-4ead-aeb5-cc56de00389f","Type":"ContainerDied","Data":"f4431a0fe49c22f8a9a008a679002d680737f5160c684e9a4783522afdf557dd"} Oct 31 01:19:24.946194 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:24.946167 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lrphq" Oct 31 01:19:25.108367 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:25.108282 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2c3c6d6-0e64-4ead-aeb5-cc56de00389f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2c3c6d6-0e64-4ead-aeb5-cc56de00389f-maas-api-openshift-ca\") pod \"f2c3c6d6-0e64-4ead-aeb5-cc56de00389f\" (UID: \"f2c3c6d6-0e64-4ead-aeb5-cc56de00389f\") " Oct 31 01:19:25.108367 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:25.108348 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2c3c6d6-0e64-4ead-aeb5-cc56de00389f-kube-api-access-9kzwq\" (UniqueName: \"kubernetes.io/projected/f2c3c6d6-0e64-4ead-aeb5-cc56de00389f-kube-api-access-9kzwq\") pod \"f2c3c6d6-0e64-4ead-aeb5-cc56de00389f\" (UID: \"f2c3c6d6-0e64-4ead-aeb5-cc56de00389f\") " Oct 31 01:19:25.108599 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:25.108386 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2c3c6d6-0e64-4ead-aeb5-cc56de00389f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2c3c6d6-0e64-4ead-aeb5-cc56de00389f-maas-api-xks-ca\") pod \"f2c3c6d6-0e64-4ead-aeb5-cc56de00389f\" (UID: \"f2c3c6d6-0e64-4ead-aeb5-cc56de00389f\") " Oct 31 01:19:25.108697 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:25.108672 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2c3c6d6-0e64-4ead-aeb5-cc56de00389f-maas-api-openshift-ca" pod "f2c3c6d6-0e64-4ead-aeb5-cc56de00389f" (UID: "f2c3c6d6-0e64-4ead-aeb5-cc56de00389f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:19:25.108832 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:25.108811 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2c3c6d6-0e64-4ead-aeb5-cc56de00389f-maas-api-openshift-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:25.110485 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:25.110455 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2c3c6d6-0e64-4ead-aeb5-cc56de00389f-maas-api-xks-ca" pod "f2c3c6d6-0e64-4ead-aeb5-cc56de00389f" (UID: "f2c3c6d6-0e64-4ead-aeb5-cc56de00389f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:19:25.110565 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:25.110465 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c3c6d6-0e64-4ead-aeb5-cc56de00389f-kube-api-access-9kzwq" pod "f2c3c6d6-0e64-4ead-aeb5-cc56de00389f" (UID: "f2c3c6d6-0e64-4ead-aeb5-cc56de00389f"). InnerVolumeSpecName "kube-api-access-9kzwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:19:25.209829 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:25.209787 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kzwq\" (UniqueName: \"kubernetes.io/projected/f2c3c6d6-0e64-4ead-aeb5-cc56de00389f-kube-api-access-9kzwq\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:25.209829 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:25.209822 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2c3c6d6-0e64-4ead-aeb5-cc56de00389f-maas-api-xks-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:25.812852 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:25.812824 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lrphq" Oct 31 01:19:25.813006 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:25.812858 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lrphq" event={"ID":"f2c3c6d6-0e64-4ead-aeb5-cc56de00389f","Type":"ContainerDied","Data":"0c6e29da0e7cd4f1acf812fe7a20c8c50e49dc294d08098c9ea62dc61221a802"} Oct 31 01:19:25.813006 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:25.812903 4440 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c6e29da0e7cd4f1acf812fe7a20c8c50e49dc294d08098c9ea62dc61221a802" Oct 31 01:19:31.000738 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:19:31.000701 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:19:32.776109 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:32.776073 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:19:36.144443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:36.142489 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-252485f4-f4rvq"] Oct 31 01:19:36.152797 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:36.152750 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-252485f4-f4rvq"] Oct 31 01:19:37.557792 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:37.557746 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:19:37.770731 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:37.770695 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f77962d-e14f-4225-b2e9-7a709dc8c4f4" path="/var/lib/kubelet/pods/7f77962d-e14f-4225-b2e9-7a709dc8c4f4/volumes" Oct 31 01:19:40.092849 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:40.092802 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lrphq"] Oct 31 01:19:40.104090 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:40.104048 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lrphq"] Oct 31 01:19:40.124749 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:40.124713 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wz4s"] Oct 31 01:19:40.125229 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:40.125215 4440 cpu_manager.go:414] "containerMap: removing container" podUID="f2c3c6d6-0e64-4ead-aeb5-cc56de00389f" containerName="revoke-keys" Oct 31 01:19:40.125276 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:40.125231 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2c3c6d6-0e64-4ead-aeb5-cc56de00389f" containerName="revoke-keys" Oct 31 01:19:40.125318 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:40.125304 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2c3c6d6-0e64-4ead-aeb5-cc56de00389f" containerName="revoke-keys" Oct 31 01:19:40.128997 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:40.128973 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wz4s" Oct 31 01:19:40.133260 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:40.133235 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zs6l6\"" Oct 31 01:19:40.138443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:40.138396 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:19:40.146671 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:40.146643 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:19:40.146870 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:40.146807 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:19:40.149014 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:40.148987 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wz4s"] Oct 31 01:19:40.208133 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:40.208096 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8wz4s\" (UID: \"f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wz4s" Oct 31 01:19:40.208322 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:40.208149 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbxjh\" (UniqueName: \"kubernetes.io/projected/f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0-kube-api-access-sbxjh\") pod \"maas-api-revoke-sub-3d4156f1-8wz4s\" (UID: \"f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wz4s" Oct 31 01:19:40.208322 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:40.208171 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8wz4s\" (UID: \"f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wz4s" Oct 31 01:19:40.309593 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:40.309554 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8wz4s\" (UID: \"f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wz4s" Oct 31 01:19:40.309783 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:40.309610 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbxjh\" (UniqueName: \"kubernetes.io/projected/f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0-kube-api-access-sbxjh\") pod \"maas-api-revoke-sub-3d4156f1-8wz4s\" (UID: \"f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wz4s" Oct 31 01:19:40.309783 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:40.309734 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8wz4s\" (UID: \"f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wz4s" Oct 31 01:19:40.310380 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:40.310346 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8wz4s\" (UID: \"f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wz4s" Oct 31 01:19:40.312285 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:40.312265 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8wz4s\" (UID: \"f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wz4s" Oct 31 01:19:40.317707 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:40.317684 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbxjh\" (UniqueName: \"kubernetes.io/projected/f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0-kube-api-access-sbxjh\") pod \"maas-api-revoke-sub-3d4156f1-8wz4s\" (UID: \"f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wz4s" Oct 31 01:19:40.442955 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:40.442919 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wz4s" Oct 31 01:19:40.725196 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:40.725166 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wz4s"] Oct 31 01:19:40.726258 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:19:40.726231 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8a376d9_78e4_4f0f_9d63_b66c5f0fb0a0.slice/crio-1242bfdea60b5b3a7ab0c8deefab8a2bdc96627ac98c6d29660f0413f34c8969 WatchSource:0}: Error finding container 1242bfdea60b5b3a7ab0c8deefab8a2bdc96627ac98c6d29660f0413f34c8969: Status 404 returned error can't find the container with id 1242bfdea60b5b3a7ab0c8deefab8a2bdc96627ac98c6d29660f0413f34c8969 Oct 31 01:19:40.940228 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:40.940190 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wz4s" event={"ID":"f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0","Type":"ContainerStarted","Data":"1242bfdea60b5b3a7ab0c8deefab8a2bdc96627ac98c6d29660f0413f34c8969"} Oct 31 01:19:41.770935 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:41.770850 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2c3c6d6-0e64-4ead-aeb5-cc56de00389f" path="/var/lib/kubelet/pods/f2c3c6d6-0e64-4ead-aeb5-cc56de00389f/volumes" Oct 31 01:19:41.947987 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:41.947953 4440 generic.go:356] "Generic (PLEG): container finished" podID="f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0" containerID="a53d3ff1d982ef5c049bb2c8bd2418cf486c94666733c6f4445746c6396297e2" exitCode=0 Oct 31 01:19:41.948172 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:41.948041 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wz4s" event={"ID":"f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0","Type":"ContainerDied","Data":"a53d3ff1d982ef5c049bb2c8bd2418cf486c94666733c6f4445746c6396297e2"} Oct 31 01:19:43.099937 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:43.099912 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wz4s" Oct 31 01:19:43.135646 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:43.135613 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0-maas-api-openshift-ca\") pod \"f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0\" (UID: \"f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0\") " Oct 31 01:19:43.135803 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:43.135737 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0-kube-api-access-sbxjh\" (UniqueName: \"kubernetes.io/projected/f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0-kube-api-access-sbxjh\") pod \"f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0\" (UID: \"f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0\") " Oct 31 01:19:43.135803 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:43.135775 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0-maas-api-xks-ca\") pod \"f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0\" (UID: \"f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0\") " Oct 31 01:19:43.136065 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:43.136038 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0-maas-api-openshift-ca" pod "f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0" (UID: "f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:19:43.137902 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:43.137868 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0-maas-api-xks-ca" pod "f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0" (UID: "f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:19:43.138022 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:43.138006 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0-kube-api-access-sbxjh" pod "f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0" (UID: "f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0"). InnerVolumeSpecName "kube-api-access-sbxjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:19:43.236923 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:43.236855 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0-maas-api-openshift-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:43.236923 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:43.236884 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbxjh\" (UniqueName: \"kubernetes.io/projected/f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0-kube-api-access-sbxjh\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:43.236923 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:43.236895 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0-maas-api-xks-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:43.969366 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:43.969333 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wz4s" Oct 31 01:19:43.969564 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:43.969363 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wz4s" event={"ID":"f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0","Type":"ContainerDied","Data":"1242bfdea60b5b3a7ab0c8deefab8a2bdc96627ac98c6d29660f0413f34c8969"} Oct 31 01:19:43.969564 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:43.969455 4440 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1242bfdea60b5b3a7ab0c8deefab8a2bdc96627ac98c6d29660f0413f34c8969" Oct 31 01:19:48.236826 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:48.236794 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:48.346394 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:48.346359 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:50.099839 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:50.099798 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:51.571585 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:51.571544 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:19:55.388995 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:55.388954 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:19:56.582563 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:56.582480 4440 scope.go:122] "RemoveContainer" containerID="005b253fdcf97c9b9d34ee752cdd302bf41717ed5dcc3cd5a3130f9253da1ada" Oct 31 01:19:56.592548 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:19:56.592523 4440 scope.go:122] "RemoveContainer" containerID="7e9c4a7ae854dee3a08c0a09f9e8767db889c6b01e7efb79faa5c8464e615674" Oct 31 01:20:01.015701 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:20:00.999940 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:20:01.320585 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:01.320495 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:20:04.535249 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:04.534501 4440 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6rjmk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:20:04.535249 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:04.534577 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" podUID="dfa83ced-0142-4d9a-9706-7edb28fda974" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:20:08.641800 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:08.641759 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-t29mq"] Oct 31 01:20:08.642181 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:08.642167 4440 cpu_manager.go:414] "containerMap: removing container" podUID="f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0" containerName="revoke-keys" Oct 31 01:20:08.642181 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:08.642177 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0" containerName="revoke-keys" Oct 31 01:20:08.642309 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:08.642288 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0" containerName="revoke-keys" Oct 31 01:20:08.646875 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:08.646854 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-t29mq" Oct 31 01:20:08.651537 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:08.651299 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:20:08.651537 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:08.651366 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:20:08.651537 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:08.651373 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:20:08.651537 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:08.651505 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zs6l6\"" Oct 31 01:20:08.654879 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:08.654853 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-t29mq"] Oct 31 01:20:08.759506 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:08.759466 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e63f5103-aba8-4617-b467-71bf70b5d456-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-t29mq\" (UID: \"e63f5103-aba8-4617-b467-71bf70b5d456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-t29mq" Oct 31 01:20:08.759706 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:08.759621 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g25s\" (UniqueName: \"kubernetes.io/projected/e63f5103-aba8-4617-b467-71bf70b5d456-kube-api-access-9g25s\") pod \"maas-api-revoke-sub-e2fc9c5b-t29mq\" (UID: \"e63f5103-aba8-4617-b467-71bf70b5d456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-t29mq" Oct 31 01:20:08.759780 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:08.759749 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e63f5103-aba8-4617-b467-71bf70b5d456-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-t29mq\" (UID: \"e63f5103-aba8-4617-b467-71bf70b5d456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-t29mq" Oct 31 01:20:08.860922 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:08.860887 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e63f5103-aba8-4617-b467-71bf70b5d456-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-t29mq\" (UID: \"e63f5103-aba8-4617-b467-71bf70b5d456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-t29mq" Oct 31 01:20:08.861100 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:08.860975 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g25s\" (UniqueName: \"kubernetes.io/projected/e63f5103-aba8-4617-b467-71bf70b5d456-kube-api-access-9g25s\") pod \"maas-api-revoke-sub-e2fc9c5b-t29mq\" (UID: \"e63f5103-aba8-4617-b467-71bf70b5d456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-t29mq" Oct 31 01:20:08.861100 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:08.861022 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e63f5103-aba8-4617-b467-71bf70b5d456-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-t29mq\" (UID: \"e63f5103-aba8-4617-b467-71bf70b5d456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-t29mq" Oct 31 01:20:08.861553 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:08.861532 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e63f5103-aba8-4617-b467-71bf70b5d456-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-t29mq\" (UID: \"e63f5103-aba8-4617-b467-71bf70b5d456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-t29mq" Oct 31 01:20:08.863317 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:08.863298 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e63f5103-aba8-4617-b467-71bf70b5d456-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-t29mq\" (UID: \"e63f5103-aba8-4617-b467-71bf70b5d456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-t29mq" Oct 31 01:20:08.869940 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:08.869915 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g25s\" (UniqueName: \"kubernetes.io/projected/e63f5103-aba8-4617-b467-71bf70b5d456-kube-api-access-9g25s\") pod \"maas-api-revoke-sub-e2fc9c5b-t29mq\" (UID: \"e63f5103-aba8-4617-b467-71bf70b5d456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-t29mq" Oct 31 01:20:08.963285 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:08.963258 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-t29mq" Oct 31 01:20:09.244780 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:09.244754 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-t29mq"] Oct 31 01:20:09.246300 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:20:09.246270 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode63f5103_aba8_4617_b467_71bf70b5d456.slice/crio-96371cd9d818c3ccf83593f53c5c39fda7d5bc341aee2356f346b97d57173364 WatchSource:0}: Error finding container 96371cd9d818c3ccf83593f53c5c39fda7d5bc341aee2356f346b97d57173364: Status 404 returned error can't find the container with id 96371cd9d818c3ccf83593f53c5c39fda7d5bc341aee2356f346b97d57173364 Oct 31 01:20:10.172881 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:10.172843 4440 generic.go:356] "Generic (PLEG): container finished" podID="e63f5103-aba8-4617-b467-71bf70b5d456" containerID="51e0fc86ffd801c0edcaae8bc1a9dbb428eeb944031826e32c4fa195b1ad91c1" exitCode=0 Oct 31 01:20:10.173288 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:10.172921 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-t29mq" event={"ID":"e63f5103-aba8-4617-b467-71bf70b5d456","Type":"ContainerDied","Data":"51e0fc86ffd801c0edcaae8bc1a9dbb428eeb944031826e32c4fa195b1ad91c1"} Oct 31 01:20:10.173288 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:10.173005 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-t29mq" event={"ID":"e63f5103-aba8-4617-b467-71bf70b5d456","Type":"ContainerStarted","Data":"96371cd9d818c3ccf83593f53c5c39fda7d5bc341aee2356f346b97d57173364"} Oct 31 01:20:10.376128 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:10.376034 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:11.317073 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:11.317050 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-t29mq" Oct 31 01:20:11.486731 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:11.486652 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e63f5103-aba8-4617-b467-71bf70b5d456-kube-api-access-9g25s\" (UniqueName: \"kubernetes.io/projected/e63f5103-aba8-4617-b467-71bf70b5d456-kube-api-access-9g25s\") pod \"e63f5103-aba8-4617-b467-71bf70b5d456\" (UID: \"e63f5103-aba8-4617-b467-71bf70b5d456\") " Oct 31 01:20:11.486910 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:11.486784 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e63f5103-aba8-4617-b467-71bf70b5d456-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e63f5103-aba8-4617-b467-71bf70b5d456-maas-api-xks-ca\") pod \"e63f5103-aba8-4617-b467-71bf70b5d456\" (UID: \"e63f5103-aba8-4617-b467-71bf70b5d456\") " Oct 31 01:20:11.486910 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:11.486813 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e63f5103-aba8-4617-b467-71bf70b5d456-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e63f5103-aba8-4617-b467-71bf70b5d456-maas-api-openshift-ca\") pod \"e63f5103-aba8-4617-b467-71bf70b5d456\" (UID: \"e63f5103-aba8-4617-b467-71bf70b5d456\") " Oct 31 01:20:11.487212 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:11.487187 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e63f5103-aba8-4617-b467-71bf70b5d456-maas-api-openshift-ca" pod "e63f5103-aba8-4617-b467-71bf70b5d456" (UID: "e63f5103-aba8-4617-b467-71bf70b5d456"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:20:11.488832 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:11.488808 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e63f5103-aba8-4617-b467-71bf70b5d456-maas-api-xks-ca" pod "e63f5103-aba8-4617-b467-71bf70b5d456" (UID: "e63f5103-aba8-4617-b467-71bf70b5d456"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:20:11.488832 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:11.488812 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e63f5103-aba8-4617-b467-71bf70b5d456-kube-api-access-9g25s" pod "e63f5103-aba8-4617-b467-71bf70b5d456" (UID: "e63f5103-aba8-4617-b467-71bf70b5d456"). InnerVolumeSpecName "kube-api-access-9g25s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:11.588387 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:11.588356 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e63f5103-aba8-4617-b467-71bf70b5d456-maas-api-xks-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:11.588387 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:11.588382 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e63f5103-aba8-4617-b467-71bf70b5d456-maas-api-openshift-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:11.588555 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:11.588399 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g25s\" (UniqueName: \"kubernetes.io/projected/e63f5103-aba8-4617-b467-71bf70b5d456-kube-api-access-9g25s\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:12.187716 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:12.187686 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-t29mq" Oct 31 01:20:12.187716 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:12.187711 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-t29mq" event={"ID":"e63f5103-aba8-4617-b467-71bf70b5d456","Type":"ContainerDied","Data":"96371cd9d818c3ccf83593f53c5c39fda7d5bc341aee2356f346b97d57173364"} Oct 31 01:20:12.187927 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:12.187753 4440 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96371cd9d818c3ccf83593f53c5c39fda7d5bc341aee2356f346b97d57173364" Oct 31 01:20:17.997507 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:17.997465 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkx55"] Oct 31 01:20:17.997894 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:17.997676 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkx55" podUID="4f77d13d-b9ab-44cd-ba25-37f8841cf8da" containerName="discovery" containerID="cri-o://c9c567a30c45fafc6d3e24ed529167e2d7f85125456380f100d331f16fa7c756" gracePeriod=30 Oct 31 01:20:18.227313 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:18.227277 4440 generic.go:356] "Generic (PLEG): container finished" podID="4f77d13d-b9ab-44cd-ba25-37f8841cf8da" containerID="c9c567a30c45fafc6d3e24ed529167e2d7f85125456380f100d331f16fa7c756" exitCode=0 Oct 31 01:20:18.227517 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:18.227359 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkx55" event={"ID":"4f77d13d-b9ab-44cd-ba25-37f8841cf8da","Type":"ContainerDied","Data":"c9c567a30c45fafc6d3e24ed529167e2d7f85125456380f100d331f16fa7c756"} Oct 31 01:20:18.257479 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:18.257411 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkx55" Oct 31 01:20:18.350194 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:18.350167 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-istio-csr-dns-cert\") pod \"4f77d13d-b9ab-44cd-ba25-37f8841cf8da\" (UID: \"4f77d13d-b9ab-44cd-ba25-37f8841cf8da\") " Oct 31 01:20:18.350194 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:18.350197 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-istio-kubeconfig\") pod \"4f77d13d-b9ab-44cd-ba25-37f8841cf8da\" (UID: \"4f77d13d-b9ab-44cd-ba25-37f8841cf8da\") " Oct 31 01:20:18.350384 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:18.350226 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-istio-csr-ca-configmap\") pod \"4f77d13d-b9ab-44cd-ba25-37f8841cf8da\" (UID: \"4f77d13d-b9ab-44cd-ba25-37f8841cf8da\") " Oct 31 01:20:18.350384 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:18.350255 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-istio-token\" (UniqueName: \"kubernetes.io/projected/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-istio-token\") pod \"4f77d13d-b9ab-44cd-ba25-37f8841cf8da\" (UID: \"4f77d13d-b9ab-44cd-ba25-37f8841cf8da\") " Oct 31 01:20:18.350384 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:18.350366 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-cacerts\" (UniqueName: \"kubernetes.io/secret/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-cacerts\") pod \"4f77d13d-b9ab-44cd-ba25-37f8841cf8da\" (UID: \"4f77d13d-b9ab-44cd-ba25-37f8841cf8da\") " Oct 31 01:20:18.350583 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:18.350404 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-local-certs\") pod \"4f77d13d-b9ab-44cd-ba25-37f8841cf8da\" (UID: \"4f77d13d-b9ab-44cd-ba25-37f8841cf8da\") " Oct 31 01:20:18.350583 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:18.350544 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-kube-api-access-cvm8s\" (UniqueName: \"kubernetes.io/projected/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-kube-api-access-cvm8s\") pod \"4f77d13d-b9ab-44cd-ba25-37f8841cf8da\" (UID: \"4f77d13d-b9ab-44cd-ba25-37f8841cf8da\") " Oct 31 01:20:18.350735 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:18.350707 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-istio-csr-ca-configmap" pod "4f77d13d-b9ab-44cd-ba25-37f8841cf8da" (UID: "4f77d13d-b9ab-44cd-ba25-37f8841cf8da"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:20:18.351013 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:18.350989 4440 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-istio-csr-ca-configmap\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:18.352670 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:18.352641 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-cacerts" pod "4f77d13d-b9ab-44cd-ba25-37f8841cf8da" (UID: "4f77d13d-b9ab-44cd-ba25-37f8841cf8da"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:20:18.352773 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:18.352663 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-kube-api-access-cvm8s" pod "4f77d13d-b9ab-44cd-ba25-37f8841cf8da" (UID: "4f77d13d-b9ab-44cd-ba25-37f8841cf8da"). InnerVolumeSpecName "kube-api-access-cvm8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:18.353012 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:18.352986 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-istio-csr-dns-cert" pod "4f77d13d-b9ab-44cd-ba25-37f8841cf8da" (UID: "4f77d13d-b9ab-44cd-ba25-37f8841cf8da"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:20:18.353101 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:18.353010 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-istio-kubeconfig" pod "4f77d13d-b9ab-44cd-ba25-37f8841cf8da" (UID: "4f77d13d-b9ab-44cd-ba25-37f8841cf8da"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:20:18.353101 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:18.353035 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-local-certs" pod "4f77d13d-b9ab-44cd-ba25-37f8841cf8da" (UID: "4f77d13d-b9ab-44cd-ba25-37f8841cf8da"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:18.353101 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:18.353042 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-istio-token" pod "4f77d13d-b9ab-44cd-ba25-37f8841cf8da" (UID: "4f77d13d-b9ab-44cd-ba25-37f8841cf8da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:18.451678 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:18.451649 4440 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-istio-csr-dns-cert\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:18.451678 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:18.451672 4440 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-istio-kubeconfig\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:18.451678 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:18.451681 4440 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-istio-token\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:18.451870 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:18.451690 4440 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-cacerts\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:18.451870 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:18.451699 4440 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-local-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:18.451870 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:18.451709 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvm8s\" (UniqueName: \"kubernetes.io/projected/4f77d13d-b9ab-44cd-ba25-37f8841cf8da-kube-api-access-cvm8s\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:18.896596 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:18.896565 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xtkrd"] Oct 31 01:20:18.897170 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:18.897156 4440 cpu_manager.go:414] "containerMap: removing container" podUID="4f77d13d-b9ab-44cd-ba25-37f8841cf8da" containerName="discovery" Oct 31 01:20:18.897225 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:18.897173 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f77d13d-b9ab-44cd-ba25-37f8841cf8da" containerName="discovery" Oct 31 01:20:18.897225 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:18.897214 4440 cpu_manager.go:414] "containerMap: removing container" podUID="e63f5103-aba8-4617-b467-71bf70b5d456" containerName="revoke-keys" Oct 31 01:20:18.897225 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:18.897222 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e63f5103-aba8-4617-b467-71bf70b5d456" containerName="revoke-keys" Oct 31 01:20:18.897323 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:18.897310 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="e63f5103-aba8-4617-b467-71bf70b5d456" containerName="revoke-keys" Oct 31 01:20:18.897360 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:18.897328 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f77d13d-b9ab-44cd-ba25-37f8841cf8da" containerName="discovery" Oct 31 01:20:18.901099 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:18.901082 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xtkrd" Oct 31 01:20:18.908215 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:18.908190 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:20:18.908361 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:18.908269 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:20:18.908474 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:18.908367 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:20:18.908474 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:18.908457 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zs6l6\"" Oct 31 01:20:18.911913 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:18.911885 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xtkrd"] Oct 31 01:20:18.956826 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:18.956790 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7e16c3b-a22a-4460-8499-e17f49ce062f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-xtkrd\" (UID: \"b7e16c3b-a22a-4460-8499-e17f49ce062f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xtkrd" Oct 31 01:20:18.956999 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:18.956970 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7e16c3b-a22a-4460-8499-e17f49ce062f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-xtkrd\" (UID: \"b7e16c3b-a22a-4460-8499-e17f49ce062f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xtkrd" Oct 31 01:20:18.957072 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:18.957027 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7pdc\" (UniqueName: \"kubernetes.io/projected/b7e16c3b-a22a-4460-8499-e17f49ce062f-kube-api-access-v7pdc\") pod \"maas-api-revoke-sub-5a29efb3-xtkrd\" (UID: \"b7e16c3b-a22a-4460-8499-e17f49ce062f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xtkrd" Oct 31 01:20:19.058088 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:19.058043 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7pdc\" (UniqueName: \"kubernetes.io/projected/b7e16c3b-a22a-4460-8499-e17f49ce062f-kube-api-access-v7pdc\") pod \"maas-api-revoke-sub-5a29efb3-xtkrd\" (UID: \"b7e16c3b-a22a-4460-8499-e17f49ce062f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xtkrd" Oct 31 01:20:19.058088 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:19.058093 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7e16c3b-a22a-4460-8499-e17f49ce062f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-xtkrd\" (UID: \"b7e16c3b-a22a-4460-8499-e17f49ce062f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xtkrd" Oct 31 01:20:19.058707 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:19.058249 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7e16c3b-a22a-4460-8499-e17f49ce062f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-xtkrd\" (UID: \"b7e16c3b-a22a-4460-8499-e17f49ce062f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xtkrd" Oct 31 01:20:19.058881 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:19.058856 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7e16c3b-a22a-4460-8499-e17f49ce062f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-xtkrd\" (UID: \"b7e16c3b-a22a-4460-8499-e17f49ce062f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xtkrd" Oct 31 01:20:19.060830 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:19.060801 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7e16c3b-a22a-4460-8499-e17f49ce062f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-xtkrd\" (UID: \"b7e16c3b-a22a-4460-8499-e17f49ce062f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xtkrd" Oct 31 01:20:19.068103 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:19.068075 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7pdc\" (UniqueName: \"kubernetes.io/projected/b7e16c3b-a22a-4460-8499-e17f49ce062f-kube-api-access-v7pdc\") pod \"maas-api-revoke-sub-5a29efb3-xtkrd\" (UID: \"b7e16c3b-a22a-4460-8499-e17f49ce062f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xtkrd" Oct 31 01:20:19.216665 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:19.216629 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xtkrd" Oct 31 01:20:19.237450 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:19.236566 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkx55" Oct 31 01:20:19.237450 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:19.236564 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkx55" event={"ID":"4f77d13d-b9ab-44cd-ba25-37f8841cf8da","Type":"ContainerDied","Data":"24c05155a0c2987e40437072a00e263ea6a87ac4a4f51fd2300cb9e774d613c8"} Oct 31 01:20:19.237450 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:19.236728 4440 scope.go:122] "RemoveContainer" containerID="c9c567a30c45fafc6d3e24ed529167e2d7f85125456380f100d331f16fa7c756" Oct 31 01:20:19.270128 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:19.270097 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkx55"] Oct 31 01:20:19.287886 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:19.287855 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkx55"] Oct 31 01:20:19.511820 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:19.511789 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xtkrd"] Oct 31 01:20:19.512313 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:20:19.512288 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7e16c3b_a22a_4460_8499_e17f49ce062f.slice/crio-51c2228e0f78a51b245364cb5b15305195f51fc62dcd28571b5985ddacadaebc WatchSource:0}: Error finding container 51c2228e0f78a51b245364cb5b15305195f51fc62dcd28571b5985ddacadaebc: Status 404 returned error can't find the container with id 51c2228e0f78a51b245364cb5b15305195f51fc62dcd28571b5985ddacadaebc Oct 31 01:20:19.600015 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:19.599984 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:20:19.771469 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:19.771388 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f77d13d-b9ab-44cd-ba25-37f8841cf8da" path="/var/lib/kubelet/pods/4f77d13d-b9ab-44cd-ba25-37f8841cf8da/volumes" Oct 31 01:20:20.248604 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:20.248311 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xtkrd" event={"ID":"b7e16c3b-a22a-4460-8499-e17f49ce062f","Type":"ContainerStarted","Data":"426811f9eb10f6e4c92df4fa04dcf1ae1a9c9b082602ea9184b8c0c7a57f3596"} Oct 31 01:20:20.248604 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:20.248378 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xtkrd" event={"ID":"b7e16c3b-a22a-4460-8499-e17f49ce062f","Type":"ContainerStarted","Data":"51c2228e0f78a51b245364cb5b15305195f51fc62dcd28571b5985ddacadaebc"} Oct 31 01:20:20.275385 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:20.275318 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xtkrd" podStartSLOduration=2.275300467 podStartE2EDuration="2.275300467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:20.272708029 +0000 UTC m=+1464.631430365" watchObservedRunningTime="2026-10-31 01:20:20.275300467 +0000 UTC m=+1464.634022783" Oct 31 01:20:20.429228 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:20:20.429191 4440 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7e16c3b_a22a_4460_8499_e17f49ce062f.slice/crio-426811f9eb10f6e4c92df4fa04dcf1ae1a9c9b082602ea9184b8c0c7a57f3596.scope\": RecentStats: unable to find data in memory cache]" Oct 31 01:20:21.272882 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:21.272843 4440 generic.go:356] "Generic (PLEG): container finished" podID="b7e16c3b-a22a-4460-8499-e17f49ce062f" containerID="426811f9eb10f6e4c92df4fa04dcf1ae1a9c9b082602ea9184b8c0c7a57f3596" exitCode=0 Oct 31 01:20:21.277593 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:21.277559 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xtkrd" event={"ID":"b7e16c3b-a22a-4460-8499-e17f49ce062f","Type":"ContainerDied","Data":"426811f9eb10f6e4c92df4fa04dcf1ae1a9c9b082602ea9184b8c0c7a57f3596"} Oct 31 01:20:22.899574 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:22.899291 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xtkrd" Oct 31 01:20:22.998429 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:22.998375 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7e16c3b-a22a-4460-8499-e17f49ce062f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7e16c3b-a22a-4460-8499-e17f49ce062f-maas-api-openshift-ca\") pod \"b7e16c3b-a22a-4460-8499-e17f49ce062f\" (UID: \"b7e16c3b-a22a-4460-8499-e17f49ce062f\") " Oct 31 01:20:22.998776 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:22.998755 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7e16c3b-a22a-4460-8499-e17f49ce062f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7e16c3b-a22a-4460-8499-e17f49ce062f-maas-api-xks-ca\") pod \"b7e16c3b-a22a-4460-8499-e17f49ce062f\" (UID: \"b7e16c3b-a22a-4460-8499-e17f49ce062f\") " Oct 31 01:20:22.998932 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:22.998917 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7e16c3b-a22a-4460-8499-e17f49ce062f-kube-api-access-v7pdc\" (UniqueName: \"kubernetes.io/projected/b7e16c3b-a22a-4460-8499-e17f49ce062f-kube-api-access-v7pdc\") pod \"b7e16c3b-a22a-4460-8499-e17f49ce062f\" (UID: \"b7e16c3b-a22a-4460-8499-e17f49ce062f\") " Oct 31 01:20:22.999154 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:22.998826 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7e16c3b-a22a-4460-8499-e17f49ce062f-maas-api-openshift-ca" pod "b7e16c3b-a22a-4460-8499-e17f49ce062f" (UID: "b7e16c3b-a22a-4460-8499-e17f49ce062f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:20:22.999474 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:22.999454 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7e16c3b-a22a-4460-8499-e17f49ce062f-maas-api-openshift-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:23.002911 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:23.001889 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7e16c3b-a22a-4460-8499-e17f49ce062f-maas-api-xks-ca" pod "b7e16c3b-a22a-4460-8499-e17f49ce062f" (UID: "b7e16c3b-a22a-4460-8499-e17f49ce062f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:20:23.003559 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:23.003301 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7e16c3b-a22a-4460-8499-e17f49ce062f-kube-api-access-v7pdc" pod "b7e16c3b-a22a-4460-8499-e17f49ce062f" (UID: "b7e16c3b-a22a-4460-8499-e17f49ce062f"). InnerVolumeSpecName "kube-api-access-v7pdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:23.100356 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:23.100328 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7e16c3b-a22a-4460-8499-e17f49ce062f-maas-api-xks-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:23.100356 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:23.100354 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7pdc\" (UniqueName: \"kubernetes.io/projected/b7e16c3b-a22a-4460-8499-e17f49ce062f-kube-api-access-v7pdc\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:23.309620 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:23.309532 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xtkrd" Oct 31 01:20:23.309620 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:23.309579 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xtkrd" event={"ID":"b7e16c3b-a22a-4460-8499-e17f49ce062f","Type":"ContainerDied","Data":"51c2228e0f78a51b245364cb5b15305195f51fc62dcd28571b5985ddacadaebc"} Oct 31 01:20:23.309802 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:23.309626 4440 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51c2228e0f78a51b245364cb5b15305195f51fc62dcd28571b5985ddacadaebc" Oct 31 01:20:29.803954 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:29.803916 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:20:31.002968 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:20:31.002926 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:20:34.855722 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:34.855686 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wz4s"] Oct 31 01:20:34.858955 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:34.858925 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wz4s"] Oct 31 01:20:35.769716 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:35.769676 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0" path="/var/lib/kubelet/pods/f8a376d9-78e4-4f0f-9d63-b66c5f0fb0a0/volumes" Oct 31 01:20:45.726926 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:45.726827 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:20:46.939443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:46.939312 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:47.316266 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:47.316171 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:49.513566 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:49.513528 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:20:53.317755 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:53.317719 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:53.632339 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:53.632255 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55"] Oct 31 01:20:53.632797 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:53.632778 4440 cpu_manager.go:414] "containerMap: removing container" podUID="b7e16c3b-a22a-4460-8499-e17f49ce062f" containerName="revoke-keys" Oct 31 01:20:53.632854 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:53.632798 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7e16c3b-a22a-4460-8499-e17f49ce062f" containerName="revoke-keys" Oct 31 01:20:53.632890 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:53.632872 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7e16c3b-a22a-4460-8499-e17f49ce062f" containerName="revoke-keys" Oct 31 01:20:53.636386 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:53.636368 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" Oct 31 01:20:53.648879 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:53.648858 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-2753ec-openshift-default-dockercfg-8rg5t\"" Oct 31 01:20:53.666943 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:53.666921 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55"] Oct 31 01:20:53.720980 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:53.720940 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-istio-envoy\") pod \"e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55\" (UID: \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\") " pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" Oct 31 01:20:53.721245 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:53.721225 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-istio-podinfo\") pod \"e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55\" (UID: \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\") " pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" Oct 31 01:20:53.721355 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:53.721342 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-istiod-ca-cert\") pod \"e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55\" (UID: \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\") " pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" Oct 31 01:20:53.721509 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:53.721493 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4gmc\" (UniqueName: \"kubernetes.io/projected/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-kube-api-access-m4gmc\") pod \"e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55\" (UID: \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\") " pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" Oct 31 01:20:53.725564 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:53.725527 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-workload-socket\") pod \"e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55\" (UID: \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\") " pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" Oct 31 01:20:53.725749 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:53.725599 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-workload-certs\") pod \"e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55\" (UID: \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\") " pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" Oct 31 01:20:53.725749 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:53.725641 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-istio-data\") pod \"e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55\" (UID: \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\") " pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" Oct 31 01:20:53.725749 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:53.725709 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-credential-socket\") pod \"e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55\" (UID: \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\") " pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" Oct 31 01:20:53.725749 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:53.725732 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-istio-token\") pod \"e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55\" (UID: \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\") " pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" Oct 31 01:20:53.749957 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:53.749922 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:53.826603 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:53.826569 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-workload-certs\") pod \"e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55\" (UID: \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\") " pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" Oct 31 01:20:53.826821 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:53.826626 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-istio-data\") pod \"e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55\" (UID: \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\") " pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" Oct 31 01:20:53.826821 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:53.826678 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-credential-socket\") pod \"e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55\" (UID: \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\") " pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" Oct 31 01:20:53.826821 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:53.826704 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-istio-token\") pod \"e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55\" (UID: \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\") " pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" Oct 31 01:20:53.826821 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:53.826752 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-istio-envoy\") pod \"e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55\" (UID: \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\") " pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" Oct 31 01:20:53.826821 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:53.826778 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-istio-podinfo\") pod \"e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55\" (UID: \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\") " pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" Oct 31 01:20:53.826821 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:53.826802 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-istiod-ca-cert\") pod \"e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55\" (UID: \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\") " pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" Oct 31 01:20:53.827172 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:53.827137 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4gmc\" (UniqueName: \"kubernetes.io/projected/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-kube-api-access-m4gmc\") pod \"e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55\" (UID: \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\") " pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" Oct 31 01:20:53.827260 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:53.827239 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-workload-socket\") pod \"e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55\" (UID: \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\") " pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" Oct 31 01:20:53.829279 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:53.827498 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-istio-data\") pod \"e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55\" (UID: \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\") " pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" Oct 31 01:20:53.829279 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:53.827608 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-workload-socket\") pod \"e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55\" (UID: \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\") " pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" Oct 31 01:20:53.829279 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:53.827652 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-workload-certs\") pod \"e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55\" (UID: \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\") " pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" Oct 31 01:20:53.829279 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:53.827724 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-istiod-ca-cert\") pod \"e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55\" (UID: \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\") " pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" Oct 31 01:20:53.829279 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:53.827978 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-credential-socket\") pod \"e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55\" (UID: \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\") " pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" Oct 31 01:20:53.833798 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:53.830501 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-istio-envoy\") pod \"e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55\" (UID: \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\") " pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" Oct 31 01:20:53.833798 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:53.830626 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-istio-podinfo\") pod \"e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55\" (UID: \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\") " pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" Oct 31 01:20:53.846010 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:53.845982 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-istio-token\") pod \"e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55\" (UID: \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\") " pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" Oct 31 01:20:53.846010 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:53.846005 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4gmc\" (UniqueName: \"kubernetes.io/projected/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-kube-api-access-m4gmc\") pod \"e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55\" (UID: \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\") " pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" Oct 31 01:20:53.939547 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:53.939516 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:53.950368 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:53.950336 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" Oct 31 01:20:54.248340 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:20:54.248307 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18fec23f_6a90_42a0_8a23_f07eaf3ab0b8.slice/crio-c114ca1e914d678bdb3ef2136a8a1eb825d017e535d6c7609f47f9b9fb62aac7 WatchSource:0}: Error finding container c114ca1e914d678bdb3ef2136a8a1eb825d017e535d6c7609f47f9b9fb62aac7: Status 404 returned error can't find the container with id c114ca1e914d678bdb3ef2136a8a1eb825d017e535d6c7609f47f9b9fb62aac7 Oct 31 01:20:54.251230 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:54.251187 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:20:54.251355 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:54.251269 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:20:54.251355 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:54.251314 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:20:54.252081 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:54.252054 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55"] Oct 31 01:20:54.563700 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:54.563607 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" event={"ID":"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8","Type":"ContainerStarted","Data":"a3713c6c20da9cd9f438c94c84ef6faffb40fccf3468c199fe30708ce388a037"} Oct 31 01:20:54.563700 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:54.563670 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" event={"ID":"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8","Type":"ContainerStarted","Data":"c114ca1e914d678bdb3ef2136a8a1eb825d017e535d6c7609f47f9b9fb62aac7"} Oct 31 01:20:54.950576 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:54.950546 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" Oct 31 01:20:55.955860 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:55.955831 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" Oct 31 01:20:56.005104 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:56.005045 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" podStartSLOduration=3.005026377 podStartE2EDuration="3.005026377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:54.589334999 +0000 UTC m=+1498.948057316" watchObservedRunningTime="2026-10-31 01:20:56.005026377 +0000 UTC m=+1500.363748697" Oct 31 01:20:56.576099 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:56.576066 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" Oct 31 01:20:56.577092 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:56.577072 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" Oct 31 01:20:56.633602 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:56.633564 4440 scope.go:122] "RemoveContainer" containerID="3068471fdc47c64e56c6a2631b1637ee76f1f1cda819149bf2c578dca9223b03" Oct 31 01:20:56.644367 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:56.644345 4440 scope.go:122] "RemoveContainer" containerID="f01cc4cde3b546e2ad2e4180a9ef3d5b748b42d08b240e74228c7e99ef4a814b" Oct 31 01:20:56.655531 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:56.655506 4440 scope.go:122] "RemoveContainer" containerID="c4555074111df94cf53b0eb4773405356efb2bf0d670e4c2f82a4f3af6b1a847" Oct 31 01:20:57.152087 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:57.152041 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:57.335770 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:57.335738 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:58.301060 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:58.301031 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2753ec-7bf7645445-jw9c4"] Oct 31 01:20:58.306520 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:58.306496 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2753ec-7bf7645445-jw9c4" Oct 31 01:20:58.322883 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:58.322850 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:20:58.323815 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:58.323796 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:20:58.323815 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:58.323797 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 01:20:58.324086 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:58.324058 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-n5p2t\"" Oct 31 01:20:58.339770 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:58.339742 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2753ec-7bf7645445-jw9c4"] Oct 31 01:20:58.370183 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:58.370155 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c4c9d10-144f-4e32-8f17-45bea8e215a3-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2753ec-7bf7645445-jw9c4\" (UID: \"4c4c9d10-144f-4e32-8f17-45bea8e215a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2753ec-7bf7645445-jw9c4" Oct 31 01:20:58.370332 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:58.370202 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c4c9d10-144f-4e32-8f17-45bea8e215a3-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2753ec-7bf7645445-jw9c4\" (UID: \"4c4c9d10-144f-4e32-8f17-45bea8e215a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2753ec-7bf7645445-jw9c4" Oct 31 01:20:58.370332 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:58.370303 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxv5t\" (UniqueName: \"kubernetes.io/projected/4c4c9d10-144f-4e32-8f17-45bea8e215a3-kube-api-access-hxv5t\") pod \"maas-api-e2e-worker-single-main-2753ec-7bf7645445-jw9c4\" (UID: \"4c4c9d10-144f-4e32-8f17-45bea8e215a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2753ec-7bf7645445-jw9c4" Oct 31 01:20:58.370433 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:58.370404 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c4c9d10-144f-4e32-8f17-45bea8e215a3-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2753ec-7bf7645445-jw9c4\" (UID: \"4c4c9d10-144f-4e32-8f17-45bea8e215a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2753ec-7bf7645445-jw9c4" Oct 31 01:20:58.406409 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:58.406378 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-2753ec-serving-cert\"" Oct 31 01:20:58.471003 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:58.470966 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c4c9d10-144f-4e32-8f17-45bea8e215a3-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2753ec-7bf7645445-jw9c4\" (UID: \"4c4c9d10-144f-4e32-8f17-45bea8e215a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2753ec-7bf7645445-jw9c4" Oct 31 01:20:58.471198 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:58.471027 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c4c9d10-144f-4e32-8f17-45bea8e215a3-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2753ec-7bf7645445-jw9c4\" (UID: \"4c4c9d10-144f-4e32-8f17-45bea8e215a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2753ec-7bf7645445-jw9c4" Oct 31 01:20:58.471198 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:58.471069 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxv5t\" (UniqueName: \"kubernetes.io/projected/4c4c9d10-144f-4e32-8f17-45bea8e215a3-kube-api-access-hxv5t\") pod \"maas-api-e2e-worker-single-main-2753ec-7bf7645445-jw9c4\" (UID: \"4c4c9d10-144f-4e32-8f17-45bea8e215a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2753ec-7bf7645445-jw9c4" Oct 31 01:20:58.471198 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:58.471088 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c4c9d10-144f-4e32-8f17-45bea8e215a3-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2753ec-7bf7645445-jw9c4\" (UID: \"4c4c9d10-144f-4e32-8f17-45bea8e215a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2753ec-7bf7645445-jw9c4" Oct 31 01:20:58.471844 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:58.471811 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c4c9d10-144f-4e32-8f17-45bea8e215a3-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2753ec-7bf7645445-jw9c4\" (UID: \"4c4c9d10-144f-4e32-8f17-45bea8e215a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2753ec-7bf7645445-jw9c4" Oct 31 01:20:58.473615 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:58.473588 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c4c9d10-144f-4e32-8f17-45bea8e215a3-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2753ec-7bf7645445-jw9c4\" (UID: \"4c4c9d10-144f-4e32-8f17-45bea8e215a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2753ec-7bf7645445-jw9c4" Oct 31 01:20:58.473738 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:58.473595 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c4c9d10-144f-4e32-8f17-45bea8e215a3-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2753ec-7bf7645445-jw9c4\" (UID: \"4c4c9d10-144f-4e32-8f17-45bea8e215a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2753ec-7bf7645445-jw9c4" Oct 31 01:20:58.501144 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:58.501121 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxv5t\" (UniqueName: \"kubernetes.io/projected/4c4c9d10-144f-4e32-8f17-45bea8e215a3-kube-api-access-hxv5t\") pod \"maas-api-e2e-worker-single-main-2753ec-7bf7645445-jw9c4\" (UID: \"4c4c9d10-144f-4e32-8f17-45bea8e215a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2753ec-7bf7645445-jw9c4" Oct 31 01:20:58.620511 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:58.620428 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2753ec-7bf7645445-jw9c4" Oct 31 01:20:58.636225 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:58.636001 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:58.917329 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:58.917306 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2753ec-7bf7645445-jw9c4"] Oct 31 01:20:58.918673 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:20:58.918640 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c4c9d10_144f_4e32_8f17_45bea8e215a3.slice/crio-aee1ec2085da3995050133518216900079e032b0fca9319d37b83370ff8af5dc WatchSource:0}: Error finding container aee1ec2085da3995050133518216900079e032b0fca9319d37b83370ff8af5dc: Status 404 returned error can't find the container with id aee1ec2085da3995050133518216900079e032b0fca9319d37b83370ff8af5dc Oct 31 01:20:58.920413 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:58.920394 4440 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 01:20:58.954858 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:58.954825 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:59.604856 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:20:59.604827 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2753ec-7bf7645445-jw9c4" event={"ID":"4c4c9d10-144f-4e32-8f17-45bea8e215a3","Type":"ContainerStarted","Data":"aee1ec2085da3995050133518216900079e032b0fca9319d37b83370ff8af5dc"} Oct 31 01:21:00.061929 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:00.061887 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-c4977"] Oct 31 01:21:00.068081 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:00.068048 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-c4977"] Oct 31 01:21:00.451125 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:00.451089 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:00.613969 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:00.613931 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2753ec-7bf7645445-jw9c4" event={"ID":"4c4c9d10-144f-4e32-8f17-45bea8e215a3","Type":"ContainerStarted","Data":"46c7a6c38557a15f51ee6fb3d52979500b347a5414143942dbcf16570e308691"} Oct 31 01:21:00.614350 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:00.614050 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2753ec-7bf7645445-jw9c4" Oct 31 01:21:00.643134 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:00.643069 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2753ec-7bf7645445-jw9c4" podStartSLOduration=1.9856789419999998 podStartE2EDuration="2.643051214s" totalImagesPullingTime="657.372272ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:21:00.640331494 +0000 UTC m=+1504.999053811" watchObservedRunningTime="2026-10-31 01:21:00.643051214 +0000 UTC m=+1505.001773531" Oct 31 01:21:00.999689 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:21:00.999652 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:21:01.372333 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:01.372235 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:01.770063 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:01.770031 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23a3f91f-a4a2-4c88-89e9-8e13f50a5416" path="/var/lib/kubelet/pods/23a3f91f-a4a2-4c88-89e9-8e13f50a5416/volumes" Oct 31 01:21:02.070193 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:02.070041 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:21:02.583937 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:02.583898 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:03.987564 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:03.987528 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:04.762607 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:04.762524 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:05.722281 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:05.722245 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:06.482336 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:06.482302 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:06.632142 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:06.632113 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2753ec-7bf7645445-jw9c4" Oct 31 01:21:07.367758 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:07.367713 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:08.311954 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:08.311917 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:09.389771 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:09.389719 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:10.221585 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:10.221513 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:11.030275 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:11.030236 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:11.577247 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:11.577223 4440 patch_prober.go:29] interesting pod/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 31 01:21:11.577402 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:11.577271 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" podUID="18fec23f-6a90-42a0-8a23-f07eaf3ab0b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 31 01:21:11.690895 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:11.690864 4440 generic.go:356] "Generic (PLEG): container finished" podID="18fec23f-6a90-42a0-8a23-f07eaf3ab0b8" containerID="a3713c6c20da9cd9f438c94c84ef6faffb40fccf3468c199fe30708ce388a037" exitCode=0 Oct 31 01:21:11.690895 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:11.690899 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" event={"ID":"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8","Type":"ContainerDied","Data":"a3713c6c20da9cd9f438c94c84ef6faffb40fccf3468c199fe30708ce388a037"} Oct 31 01:21:11.691288 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:11.691271 4440 scope.go:122] "RemoveContainer" containerID="a3713c6c20da9cd9f438c94c84ef6faffb40fccf3468c199fe30708ce388a037" Oct 31 01:21:11.692552 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:11.692527 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:21:11.692625 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:11.692599 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:21:11.692664 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:11.692648 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:21:11.751461 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:11.751377 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:11.988517 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:11.988468 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:12.714564 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:12.714499 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:12.721262 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:12.719999 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" event={"ID":"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8","Type":"ContainerStarted","Data":"5f844176526f4a4f9d20e6ba12035259140e24e5c1594b8d2e4ba28cb1cf5172"} Oct 31 01:21:12.950723 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:12.950691 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" Oct 31 01:21:12.950907 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:12.950884 4440 patch_prober.go:29] interesting pod/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 31 01:21:12.950965 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:12.950946 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" podUID="18fec23f-6a90-42a0-8a23-f07eaf3ab0b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 31 01:21:13.021482 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:13.021384 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:13.727207 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:13.727169 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:13.951033 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:13.950994 4440 patch_prober.go:29] interesting pod/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 31 01:21:13.951228 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:13.951063 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" podUID="18fec23f-6a90-42a0-8a23-f07eaf3ab0b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 31 01:21:14.430404 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:14.430365 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:14.561210 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:14.561179 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-2753ec/simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7"] Oct 31 01:21:14.565133 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:14.565116 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2753ec/simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7" Oct 31 01:21:14.583970 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:14.583952 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-2753ec\"/\"kube-root-ca.crt\"" Oct 31 01:21:14.584998 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:14.584944 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-2753ec\"/\"simulator-single-main-2753ec-kserve-self-signed-certs\"" Oct 31 01:21:14.585147 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:14.585024 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-2753ec\"/\"openshift-service-ca.crt\"" Oct 31 01:21:14.586246 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:14.586225 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-2753ec\"/\"default-dockercfg-2z4z9\"" Oct 31 01:21:14.589484 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:14.589464 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-2753ec/simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7"] Oct 31 01:21:14.651076 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:14.651044 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc93cb50-1b43-46a3-940a-f3d302743d4d-home\") pod \"simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7\" (UID: \"cc93cb50-1b43-46a3-940a-f3d302743d4d\") " pod="e2e-models-e2e-worker-single-main-2753ec/simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7" Oct 31 01:21:14.651252 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:14.651182 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcdnq\" (UniqueName: \"kubernetes.io/projected/cc93cb50-1b43-46a3-940a-f3d302743d4d-kube-api-access-xcdnq\") pod \"simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7\" (UID: \"cc93cb50-1b43-46a3-940a-f3d302743d4d\") " pod="e2e-models-e2e-worker-single-main-2753ec/simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7" Oct 31 01:21:14.651252 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:14.651228 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc93cb50-1b43-46a3-940a-f3d302743d4d-tmp-dir\") pod \"simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7\" (UID: \"cc93cb50-1b43-46a3-940a-f3d302743d4d\") " pod="e2e-models-e2e-worker-single-main-2753ec/simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7" Oct 31 01:21:14.651373 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:14.651266 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc93cb50-1b43-46a3-940a-f3d302743d4d-tls-certs\") pod \"simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7\" (UID: \"cc93cb50-1b43-46a3-940a-f3d302743d4d\") " pod="e2e-models-e2e-worker-single-main-2753ec/simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7" Oct 31 01:21:14.651472 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:14.651447 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc93cb50-1b43-46a3-940a-f3d302743d4d-dshm\") pod \"simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7\" (UID: \"cc93cb50-1b43-46a3-940a-f3d302743d4d\") " pod="e2e-models-e2e-worker-single-main-2753ec/simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7" Oct 31 01:21:14.651692 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:14.651664 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc93cb50-1b43-46a3-940a-f3d302743d4d-model-cache\") pod \"simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7\" (UID: \"cc93cb50-1b43-46a3-940a-f3d302743d4d\") " pod="e2e-models-e2e-worker-single-main-2753ec/simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7" Oct 31 01:21:14.664246 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:14.664224 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:14.752720 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:14.752629 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc93cb50-1b43-46a3-940a-f3d302743d4d-dshm\") pod \"simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7\" (UID: \"cc93cb50-1b43-46a3-940a-f3d302743d4d\") " pod="e2e-models-e2e-worker-single-main-2753ec/simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7" Oct 31 01:21:14.752720 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:14.752676 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc93cb50-1b43-46a3-940a-f3d302743d4d-model-cache\") pod \"simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7\" (UID: \"cc93cb50-1b43-46a3-940a-f3d302743d4d\") " pod="e2e-models-e2e-worker-single-main-2753ec/simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7" Oct 31 01:21:14.752720 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:14.752719 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc93cb50-1b43-46a3-940a-f3d302743d4d-home\") pod \"simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7\" (UID: \"cc93cb50-1b43-46a3-940a-f3d302743d4d\") " pod="e2e-models-e2e-worker-single-main-2753ec/simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7" Oct 31 01:21:14.753367 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:14.752775 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcdnq\" (UniqueName: \"kubernetes.io/projected/cc93cb50-1b43-46a3-940a-f3d302743d4d-kube-api-access-xcdnq\") pod \"simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7\" (UID: \"cc93cb50-1b43-46a3-940a-f3d302743d4d\") " pod="e2e-models-e2e-worker-single-main-2753ec/simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7" Oct 31 01:21:14.753367 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:14.752793 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc93cb50-1b43-46a3-940a-f3d302743d4d-tmp-dir\") pod \"simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7\" (UID: \"cc93cb50-1b43-46a3-940a-f3d302743d4d\") " pod="e2e-models-e2e-worker-single-main-2753ec/simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7" Oct 31 01:21:14.753367 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:14.752812 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc93cb50-1b43-46a3-940a-f3d302743d4d-tls-certs\") pod \"simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7\" (UID: \"cc93cb50-1b43-46a3-940a-f3d302743d4d\") " pod="e2e-models-e2e-worker-single-main-2753ec/simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7" Oct 31 01:21:14.753367 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:14.753126 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc93cb50-1b43-46a3-940a-f3d302743d4d-home\") pod \"simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7\" (UID: \"cc93cb50-1b43-46a3-940a-f3d302743d4d\") " pod="e2e-models-e2e-worker-single-main-2753ec/simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7" Oct 31 01:21:14.753367 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:14.753230 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc93cb50-1b43-46a3-940a-f3d302743d4d-tmp-dir\") pod \"simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7\" (UID: \"cc93cb50-1b43-46a3-940a-f3d302743d4d\") " pod="e2e-models-e2e-worker-single-main-2753ec/simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7" Oct 31 01:21:14.753367 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:14.753275 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc93cb50-1b43-46a3-940a-f3d302743d4d-model-cache\") pod \"simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7\" (UID: \"cc93cb50-1b43-46a3-940a-f3d302743d4d\") " pod="e2e-models-e2e-worker-single-main-2753ec/simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7" Oct 31 01:21:14.755048 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:14.755023 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc93cb50-1b43-46a3-940a-f3d302743d4d-dshm\") pod \"simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7\" (UID: \"cc93cb50-1b43-46a3-940a-f3d302743d4d\") " pod="e2e-models-e2e-worker-single-main-2753ec/simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7" Oct 31 01:21:14.755512 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:14.755489 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc93cb50-1b43-46a3-940a-f3d302743d4d-tls-certs\") pod \"simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7\" (UID: \"cc93cb50-1b43-46a3-940a-f3d302743d4d\") " pod="e2e-models-e2e-worker-single-main-2753ec/simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7" Oct 31 01:21:14.764665 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:14.764632 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcdnq\" (UniqueName: \"kubernetes.io/projected/cc93cb50-1b43-46a3-940a-f3d302743d4d-kube-api-access-xcdnq\") pod \"simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7\" (UID: \"cc93cb50-1b43-46a3-940a-f3d302743d4d\") " pod="e2e-models-e2e-worker-single-main-2753ec/simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7" Oct 31 01:21:14.877065 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:14.877029 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2753ec/simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7" Oct 31 01:21:14.951530 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:14.951493 4440 patch_prober.go:29] interesting pod/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 31 01:21:14.951692 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:14.951574 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" podUID="18fec23f-6a90-42a0-8a23-f07eaf3ab0b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 31 01:21:15.180197 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:15.180165 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:15.195338 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:15.195310 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-2753ec/simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7"] Oct 31 01:21:15.206437 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:21:15.196641 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc93cb50_1b43_46a3_940a_f3d302743d4d.slice/crio-bee3c5b5d08e6ed3a198a841c1e2bbda50a1a9c2d6a527a98dacae45263f870e WatchSource:0}: Error finding container bee3c5b5d08e6ed3a198a841c1e2bbda50a1a9c2d6a527a98dacae45263f870e: Status 404 returned error can't find the container with id bee3c5b5d08e6ed3a198a841c1e2bbda50a1a9c2d6a527a98dacae45263f870e Oct 31 01:21:15.335583 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:15.333639 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:15.741875 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:15.741840 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2753ec/simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7" event={"ID":"cc93cb50-1b43-46a3-940a-f3d302743d4d","Type":"ContainerStarted","Data":"63738305388bf240a8d74bc19be9d7708fe3340b98869ad0731a9dba53faf293"} Oct 31 01:21:15.741875 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:15.741885 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2753ec/simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7" event={"ID":"cc93cb50-1b43-46a3-940a-f3d302743d4d","Type":"ContainerStarted","Data":"bee3c5b5d08e6ed3a198a841c1e2bbda50a1a9c2d6a527a98dacae45263f870e"} Oct 31 01:21:15.763623 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:15.763573 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-2753ec/simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7" podStartSLOduration=1.763559681 podStartE2EDuration="1.763559681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:21:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:21:15.761789809 +0000 UTC m=+1520.120512126" watchObservedRunningTime="2026-10-31 01:21:15.763559681 +0000 UTC m=+1520.122281997" Oct 31 01:21:15.877908 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:15.877872 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-2753ec/simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7" Oct 31 01:21:15.951712 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:15.951674 4440 patch_prober.go:29] interesting pod/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 31 01:21:15.951890 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:15.951740 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" podUID="18fec23f-6a90-42a0-8a23-f07eaf3ab0b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 31 01:21:16.161838 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:16.161597 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:16.732815 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:16.732777 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:21:16.948412 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:16.948363 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:16.951034 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:16.950992 4440 patch_prober.go:29] interesting pod/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 31 01:21:16.951225 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:16.951199 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" podUID="18fec23f-6a90-42a0-8a23-f07eaf3ab0b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 31 01:21:17.944384 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:17.944348 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:18.156048 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:18.156016 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" Oct 31 01:21:18.156552 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:18.156470 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" Oct 31 01:21:18.157715 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:18.157681 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" Oct 31 01:21:18.909141 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:18.909107 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:19.719479 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:19.719444 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:21.049355 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:21.049312 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:21.724726 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:21.724691 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:22.646174 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:22.646139 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:23.539705 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:23.539648 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:24.324107 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:24.324072 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:24.877617 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:24.877576 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-2753ec/simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7" Oct 31 01:21:24.890270 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:24.890244 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-2753ec/simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7" Oct 31 01:21:25.092230 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:25.092193 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:25.825872 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:25.825827 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-2753ec/simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7" Oct 31 01:21:25.859048 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:25.859019 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:26.504578 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:26.504539 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:27.213077 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:27.213036 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:28.054316 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:28.054281 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:28.714196 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:28.714161 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:29.386619 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:29.386579 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:30.059700 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:30.059667 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:30.799903 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:30.799871 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:30.999953 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:21:30.999912 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:21:31.648188 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:31.648145 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:31.696776 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:31.696737 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:21:32.613737 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:32.613697 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:33.405572 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:33.405529 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:34.007306 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:34.007266 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:34.681500 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:34.681461 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:35.293087 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:35.293047 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:36.007808 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:36.007767 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:36.696434 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:36.696377 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:37.388605 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:37.388575 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:37.982720 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:37.982682 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:38.583659 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:38.583622 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:39.381051 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:39.381009 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:39.986026 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:39.985975 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:40.621933 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:40.621894 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:41.297070 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:41.297032 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:41.915622 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:41.915580 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:42.606756 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:42.606722 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:43.302588 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:43.302553 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:44.010666 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:44.010634 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:44.704720 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:44.704679 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:45.408545 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:45.408505 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:46.083258 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:46.083219 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:46.694211 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:46.694168 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:47.012484 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:47.012384 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:47.304994 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:47.304907 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:47.432453 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:47.432403 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:48.100647 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:48.100609 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:48.704117 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:48.704076 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:49.408508 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:49.408474 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:50.094099 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:50.094059 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:50.798163 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:50.798029 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:51.618901 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:51.618860 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:52.223026 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:52.222992 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:52.822038 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:52.822000 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:53.024807 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:53.024776 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dsqxl"] Oct 31 01:21:53.027790 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:53.027764 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dsqxl"] Oct 31 01:21:53.491083 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:53.491043 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:53.771551 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:53.771461 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54448a0d-f363-455a-bd78-7026acfc2401" path="/var/lib/kubelet/pods/54448a0d-f363-455a-bd78-7026acfc2401/volumes" Oct 31 01:21:54.203406 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:54.203366 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:54.875075 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:54.875042 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:54.990169 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:54.990128 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:55.688580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:55.688541 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:55.707226 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:55.707183 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:21:56.464279 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:56.464170 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:57.189779 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:57.189738 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:57.859762 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:57.859728 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:58.194314 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:58.194276 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:58.607044 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:58.606927 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:59.398712 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:59.398674 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:59.870212 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:59.870172 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-2753ec/simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7"] Oct 31 01:21:59.991091 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:21:59.991054 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:00.096035 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:00.095993 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:00.712223 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:00.712184 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:01.000716 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:22:01.000620 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:22:01.418524 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:01.418483 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:02.081008 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:02.080968 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:02.720229 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:02.720196 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:03.410431 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:03.410379 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:04.094479 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:04.094438 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:04.786062 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:04.786028 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:05.264323 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:05.264283 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:05.301296 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:05.301252 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2753ec-7bf7645445-jw9c4"] Oct 31 01:22:05.301581 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:05.301548 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2753ec-7bf7645445-jw9c4" podUID="4c4c9d10-144f-4e32-8f17-45bea8e215a3" containerName="maas-api" containerID="cri-o://46c7a6c38557a15f51ee6fb3d52979500b347a5414143942dbcf16570e308691" gracePeriod=30 Oct 31 01:22:05.380042 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:05.380001 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:05.627677 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:05.627644 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2753ec-7bf7645445-jw9c4" Oct 31 01:22:05.796078 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:05.795973 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c4c9d10-144f-4e32-8f17-45bea8e215a3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c4c9d10-144f-4e32-8f17-45bea8e215a3-metrics-certs\") pod \"4c4c9d10-144f-4e32-8f17-45bea8e215a3\" (UID: \"4c4c9d10-144f-4e32-8f17-45bea8e215a3\") " Oct 31 01:22:05.796078 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:05.796050 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c4c9d10-144f-4e32-8f17-45bea8e215a3-kube-api-access-hxv5t\" (UniqueName: \"kubernetes.io/projected/4c4c9d10-144f-4e32-8f17-45bea8e215a3-kube-api-access-hxv5t\") pod \"4c4c9d10-144f-4e32-8f17-45bea8e215a3\" (UID: \"4c4c9d10-144f-4e32-8f17-45bea8e215a3\") " Oct 31 01:22:05.796685 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:05.796119 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c4c9d10-144f-4e32-8f17-45bea8e215a3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c4c9d10-144f-4e32-8f17-45bea8e215a3-maas-api-tls\") pod \"4c4c9d10-144f-4e32-8f17-45bea8e215a3\" (UID: \"4c4c9d10-144f-4e32-8f17-45bea8e215a3\") " Oct 31 01:22:05.796685 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:05.796248 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c4c9d10-144f-4e32-8f17-45bea8e215a3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c4c9d10-144f-4e32-8f17-45bea8e215a3-openshift-service-ca\") pod \"4c4c9d10-144f-4e32-8f17-45bea8e215a3\" (UID: \"4c4c9d10-144f-4e32-8f17-45bea8e215a3\") " Oct 31 01:22:05.796799 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:05.796677 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c4c9d10-144f-4e32-8f17-45bea8e215a3-openshift-service-ca" pod "4c4c9d10-144f-4e32-8f17-45bea8e215a3" (UID: "4c4c9d10-144f-4e32-8f17-45bea8e215a3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:22:05.798363 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:05.798322 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c4c9d10-144f-4e32-8f17-45bea8e215a3-metrics-certs" pod "4c4c9d10-144f-4e32-8f17-45bea8e215a3" (UID: "4c4c9d10-144f-4e32-8f17-45bea8e215a3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:22:05.798801 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:05.798774 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c4c9d10-144f-4e32-8f17-45bea8e215a3-maas-api-tls" pod "4c4c9d10-144f-4e32-8f17-45bea8e215a3" (UID: "4c4c9d10-144f-4e32-8f17-45bea8e215a3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:22:05.798863 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:05.798780 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c4c9d10-144f-4e32-8f17-45bea8e215a3-kube-api-access-hxv5t" pod "4c4c9d10-144f-4e32-8f17-45bea8e215a3" (UID: "4c4c9d10-144f-4e32-8f17-45bea8e215a3"). InnerVolumeSpecName "kube-api-access-hxv5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:05.897882 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:05.897845 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c4c9d10-144f-4e32-8f17-45bea8e215a3-maas-api-tls\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:05.897882 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:05.897879 4440 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c4c9d10-144f-4e32-8f17-45bea8e215a3-openshift-service-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:05.897882 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:05.897888 4440 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c4c9d10-144f-4e32-8f17-45bea8e215a3-metrics-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:05.898181 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:05.897897 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxv5t\" (UniqueName: \"kubernetes.io/projected/4c4c9d10-144f-4e32-8f17-45bea8e215a3-kube-api-access-hxv5t\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:06.094637 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:06.094541 4440 generic.go:356] "Generic (PLEG): container finished" podID="4c4c9d10-144f-4e32-8f17-45bea8e215a3" containerID="46c7a6c38557a15f51ee6fb3d52979500b347a5414143942dbcf16570e308691" exitCode=0 Oct 31 01:22:06.094637 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:06.094599 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2753ec-7bf7645445-jw9c4" Oct 31 01:22:06.094889 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:06.094624 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2753ec-7bf7645445-jw9c4" event={"ID":"4c4c9d10-144f-4e32-8f17-45bea8e215a3","Type":"ContainerDied","Data":"46c7a6c38557a15f51ee6fb3d52979500b347a5414143942dbcf16570e308691"} Oct 31 01:22:06.094889 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:06.094692 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2753ec-7bf7645445-jw9c4" event={"ID":"4c4c9d10-144f-4e32-8f17-45bea8e215a3","Type":"ContainerDied","Data":"aee1ec2085da3995050133518216900079e032b0fca9319d37b83370ff8af5dc"} Oct 31 01:22:06.094889 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:06.094724 4440 scope.go:122] "RemoveContainer" containerID="46c7a6c38557a15f51ee6fb3d52979500b347a5414143942dbcf16570e308691" Oct 31 01:22:06.105772 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:06.105754 4440 scope.go:122] "RemoveContainer" containerID="46c7a6c38557a15f51ee6fb3d52979500b347a5414143942dbcf16570e308691" Oct 31 01:22:06.106062 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:22:06.106042 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46c7a6c38557a15f51ee6fb3d52979500b347a5414143942dbcf16570e308691\": container with ID starting with 46c7a6c38557a15f51ee6fb3d52979500b347a5414143942dbcf16570e308691 not found: ID does not exist" containerID="46c7a6c38557a15f51ee6fb3d52979500b347a5414143942dbcf16570e308691" Oct 31 01:22:06.106113 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:06.106072 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46c7a6c38557a15f51ee6fb3d52979500b347a5414143942dbcf16570e308691"} err="failed to get container status \"46c7a6c38557a15f51ee6fb3d52979500b347a5414143942dbcf16570e308691\": rpc error: code = NotFound desc = could not find container \"46c7a6c38557a15f51ee6fb3d52979500b347a5414143942dbcf16570e308691\": container with ID starting with 46c7a6c38557a15f51ee6fb3d52979500b347a5414143942dbcf16570e308691 not found: ID does not exist" Oct 31 01:22:06.119838 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:06.119190 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:06.120846 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:06.120818 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2753ec-7bf7645445-jw9c4"] Oct 31 01:22:06.123834 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:06.123804 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2753ec-7bf7645445-jw9c4"] Oct 31 01:22:06.235861 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:06.235829 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:06.636116 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:06.636073 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:07.686854 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:07.686818 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:22:07.774316 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:07.774280 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c4c9d10-144f-4e32-8f17-45bea8e215a3" path="/var/lib/kubelet/pods/4c4c9d10-144f-4e32-8f17-45bea8e215a3/volumes" Oct 31 01:22:12.024611 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:12.024571 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g9f7d"] Oct 31 01:22:12.027460 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:12.027439 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g9f7d"] Oct 31 01:22:12.672021 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:12.671983 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:12.796757 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:12.796710 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:12.966090 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:12.966003 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55"] Oct 31 01:22:12.966351 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:12.966293 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" podUID="18fec23f-6a90-42a0-8a23-f07eaf3ab0b8" containerName="istio-proxy" containerID="cri-o://5f844176526f4a4f9d20e6ba12035259140e24e5c1594b8d2e4ba28cb1cf5172" gracePeriod=30 Oct 31 01:22:13.014221 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:13.014182 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:13.772568 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:13.772535 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2" path="/var/lib/kubelet/pods/e367eb3c-bee2-45ee-9b8f-1d3bb4f78ed2/volumes" Oct 31 01:22:14.891117 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:14.891078 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-2753ec/simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7" podUID="cc93cb50-1b43-46a3-940a-f3d302743d4d" containerName="main" containerID="cri-o://63738305388bf240a8d74bc19be9d7708fe3340b98869ad0731a9dba53faf293" gracePeriod=45 Oct 31 01:22:15.152726 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:15.152653 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2753ec/simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7" Oct 31 01:22:15.159092 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:15.159059 4440 generic.go:356] "Generic (PLEG): container finished" podID="cc93cb50-1b43-46a3-940a-f3d302743d4d" containerID="63738305388bf240a8d74bc19be9d7708fe3340b98869ad0731a9dba53faf293" exitCode=0 Oct 31 01:22:15.159247 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:15.159113 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2753ec/simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7" Oct 31 01:22:15.159247 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:15.159145 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2753ec/simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7" event={"ID":"cc93cb50-1b43-46a3-940a-f3d302743d4d","Type":"ContainerDied","Data":"63738305388bf240a8d74bc19be9d7708fe3340b98869ad0731a9dba53faf293"} Oct 31 01:22:15.159247 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:15.159194 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2753ec/simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7" event={"ID":"cc93cb50-1b43-46a3-940a-f3d302743d4d","Type":"ContainerDied","Data":"bee3c5b5d08e6ed3a198a841c1e2bbda50a1a9c2d6a527a98dacae45263f870e"} Oct 31 01:22:15.159247 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:15.159222 4440 scope.go:122] "RemoveContainer" containerID="63738305388bf240a8d74bc19be9d7708fe3340b98869ad0731a9dba53faf293" Oct 31 01:22:15.169104 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:15.169081 4440 scope.go:122] "RemoveContainer" containerID="63738305388bf240a8d74bc19be9d7708fe3340b98869ad0731a9dba53faf293" Oct 31 01:22:15.169604 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:22:15.169571 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63738305388bf240a8d74bc19be9d7708fe3340b98869ad0731a9dba53faf293\": container with ID starting with 63738305388bf240a8d74bc19be9d7708fe3340b98869ad0731a9dba53faf293 not found: ID does not exist" containerID="63738305388bf240a8d74bc19be9d7708fe3340b98869ad0731a9dba53faf293" Oct 31 01:22:15.169752 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:15.169619 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63738305388bf240a8d74bc19be9d7708fe3340b98869ad0731a9dba53faf293"} err="failed to get container status \"63738305388bf240a8d74bc19be9d7708fe3340b98869ad0731a9dba53faf293\": rpc error: code = NotFound desc = could not find container \"63738305388bf240a8d74bc19be9d7708fe3340b98869ad0731a9dba53faf293\": container with ID starting with 63738305388bf240a8d74bc19be9d7708fe3340b98869ad0731a9dba53faf293 not found: ID does not exist" Oct 31 01:22:15.212069 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:15.212035 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc93cb50-1b43-46a3-940a-f3d302743d4d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc93cb50-1b43-46a3-940a-f3d302743d4d-dshm\") pod \"cc93cb50-1b43-46a3-940a-f3d302743d4d\" (UID: \"cc93cb50-1b43-46a3-940a-f3d302743d4d\") " Oct 31 01:22:15.212280 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:15.212095 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc93cb50-1b43-46a3-940a-f3d302743d4d-kube-api-access-xcdnq\" (UniqueName: \"kubernetes.io/projected/cc93cb50-1b43-46a3-940a-f3d302743d4d-kube-api-access-xcdnq\") pod \"cc93cb50-1b43-46a3-940a-f3d302743d4d\" (UID: \"cc93cb50-1b43-46a3-940a-f3d302743d4d\") " Oct 31 01:22:15.212280 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:15.212149 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc93cb50-1b43-46a3-940a-f3d302743d4d-home\" (UniqueName: \"kubernetes.io/empty-dir/cc93cb50-1b43-46a3-940a-f3d302743d4d-home\") pod \"cc93cb50-1b43-46a3-940a-f3d302743d4d\" (UID: \"cc93cb50-1b43-46a3-940a-f3d302743d4d\") " Oct 31 01:22:15.212280 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:15.212176 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc93cb50-1b43-46a3-940a-f3d302743d4d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc93cb50-1b43-46a3-940a-f3d302743d4d-tmp-dir\") pod \"cc93cb50-1b43-46a3-940a-f3d302743d4d\" (UID: \"cc93cb50-1b43-46a3-940a-f3d302743d4d\") " Oct 31 01:22:15.212538 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:15.212319 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc93cb50-1b43-46a3-940a-f3d302743d4d-tls-certs\" (UniqueName: \"kubernetes.io/secret/cc93cb50-1b43-46a3-940a-f3d302743d4d-tls-certs\") pod \"cc93cb50-1b43-46a3-940a-f3d302743d4d\" (UID: \"cc93cb50-1b43-46a3-940a-f3d302743d4d\") " Oct 31 01:22:15.212538 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:15.212344 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc93cb50-1b43-46a3-940a-f3d302743d4d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc93cb50-1b43-46a3-940a-f3d302743d4d-model-cache\") pod \"cc93cb50-1b43-46a3-940a-f3d302743d4d\" (UID: \"cc93cb50-1b43-46a3-940a-f3d302743d4d\") " Oct 31 01:22:15.212966 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:15.212907 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc93cb50-1b43-46a3-940a-f3d302743d4d-home" pod "cc93cb50-1b43-46a3-940a-f3d302743d4d" (UID: "cc93cb50-1b43-46a3-940a-f3d302743d4d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:15.212966 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:15.212923 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc93cb50-1b43-46a3-940a-f3d302743d4d-tmp-dir" pod "cc93cb50-1b43-46a3-940a-f3d302743d4d" (UID: "cc93cb50-1b43-46a3-940a-f3d302743d4d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:15.213540 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:15.213510 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc93cb50-1b43-46a3-940a-f3d302743d4d-model-cache" pod "cc93cb50-1b43-46a3-940a-f3d302743d4d" (UID: "cc93cb50-1b43-46a3-940a-f3d302743d4d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:15.214858 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:15.214830 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc93cb50-1b43-46a3-940a-f3d302743d4d-dshm" pod "cc93cb50-1b43-46a3-940a-f3d302743d4d" (UID: "cc93cb50-1b43-46a3-940a-f3d302743d4d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:15.215119 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:15.215097 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc93cb50-1b43-46a3-940a-f3d302743d4d-tls-certs" pod "cc93cb50-1b43-46a3-940a-f3d302743d4d" (UID: "cc93cb50-1b43-46a3-940a-f3d302743d4d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:22:15.215254 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:15.215235 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc93cb50-1b43-46a3-940a-f3d302743d4d-kube-api-access-xcdnq" pod "cc93cb50-1b43-46a3-940a-f3d302743d4d" (UID: "cc93cb50-1b43-46a3-940a-f3d302743d4d"). InnerVolumeSpecName "kube-api-access-xcdnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:15.313940 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:15.313901 4440 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc93cb50-1b43-46a3-940a-f3d302743d4d-tls-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:15.313940 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:15.313937 4440 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc93cb50-1b43-46a3-940a-f3d302743d4d-model-cache\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:15.313940 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:15.313947 4440 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc93cb50-1b43-46a3-940a-f3d302743d4d-dshm\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:15.314185 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:15.313958 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcdnq\" (UniqueName: \"kubernetes.io/projected/cc93cb50-1b43-46a3-940a-f3d302743d4d-kube-api-access-xcdnq\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:15.314185 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:15.313968 4440 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc93cb50-1b43-46a3-940a-f3d302743d4d-home\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:15.314185 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:15.313976 4440 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc93cb50-1b43-46a3-940a-f3d302743d4d-tmp-dir\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:15.483851 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:15.483812 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-2753ec/simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7"] Oct 31 01:22:15.486826 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:15.486795 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-2753ec/simulator-single-main-2753ec-kserve-7fdb9b86c7-l9sb7"] Oct 31 01:22:15.770739 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:15.770652 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc93cb50-1b43-46a3-940a-f3d302743d4d" path="/var/lib/kubelet/pods/cc93cb50-1b43-46a3-940a-f3d302743d4d/volumes" Oct 31 01:22:18.185104 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:18.185071 4440 generic.go:356] "Generic (PLEG): container finished" podID="18fec23f-6a90-42a0-8a23-f07eaf3ab0b8" containerID="5f844176526f4a4f9d20e6ba12035259140e24e5c1594b8d2e4ba28cb1cf5172" exitCode=0 Oct 31 01:22:18.185644 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:18.185145 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" event={"ID":"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8","Type":"ContainerDied","Data":"5f844176526f4a4f9d20e6ba12035259140e24e5c1594b8d2e4ba28cb1cf5172"} Oct 31 01:22:18.185644 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:18.185213 4440 scope.go:122] "RemoveContainer" containerID="a3713c6c20da9cd9f438c94c84ef6faffb40fccf3468c199fe30708ce388a037" Oct 31 01:22:18.226064 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:18.226030 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" Oct 31 01:22:18.347290 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:18.347192 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-istio-data\") pod \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\" (UID: \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\") " Oct 31 01:22:18.347290 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:18.347284 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-istio-token\" (UniqueName: \"kubernetes.io/projected/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-istio-token\") pod \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\" (UID: \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\") " Oct 31 01:22:18.347572 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:18.347307 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-workload-certs\") pod \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\" (UID: \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\") " Oct 31 01:22:18.347572 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:18.347342 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-istio-podinfo\") pod \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\" (UID: \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\") " Oct 31 01:22:18.347572 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:18.347381 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-credential-socket\") pod \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\" (UID: \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\") " Oct 31 01:22:18.347572 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:18.347413 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-workload-socket\") pod \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\" (UID: \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\") " Oct 31 01:22:18.347572 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:18.347517 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-istio-envoy\") pod \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\" (UID: \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\") " Oct 31 01:22:18.347572 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:18.347553 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-istio-data" pod "18fec23f-6a90-42a0-8a23-f07eaf3ab0b8" (UID: "18fec23f-6a90-42a0-8a23-f07eaf3ab0b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:18.347906 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:18.347561 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-kube-api-access-m4gmc\" (UniqueName: \"kubernetes.io/projected/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-kube-api-access-m4gmc\") pod \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\" (UID: \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\") " Oct 31 01:22:18.347906 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:18.347617 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-workload-certs" pod "18fec23f-6a90-42a0-8a23-f07eaf3ab0b8" (UID: "18fec23f-6a90-42a0-8a23-f07eaf3ab0b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:18.347906 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:18.347630 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-istiod-ca-cert\") pod \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\" (UID: \"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8\") " Oct 31 01:22:18.347906 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:18.347802 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-workload-socket" pod "18fec23f-6a90-42a0-8a23-f07eaf3ab0b8" (UID: "18fec23f-6a90-42a0-8a23-f07eaf3ab0b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:18.348129 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:18.347976 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-istiod-ca-cert" pod "18fec23f-6a90-42a0-8a23-f07eaf3ab0b8" (UID: "18fec23f-6a90-42a0-8a23-f07eaf3ab0b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:22:18.348129 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:18.348119 4440 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-istio-data\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:18.348245 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:18.348140 4440 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-workload-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:18.348245 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:18.348154 4440 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-workload-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:18.348245 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:18.348163 4440 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-istiod-ca-cert\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:18.348245 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:18.348112 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-credential-socket" pod "18fec23f-6a90-42a0-8a23-f07eaf3ab0b8" (UID: "18fec23f-6a90-42a0-8a23-f07eaf3ab0b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:18.349784 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:18.349757 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-kube-api-access-m4gmc" pod "18fec23f-6a90-42a0-8a23-f07eaf3ab0b8" (UID: "18fec23f-6a90-42a0-8a23-f07eaf3ab0b8"). InnerVolumeSpecName "kube-api-access-m4gmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:18.349884 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:18.349795 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-istio-podinfo" pod "18fec23f-6a90-42a0-8a23-f07eaf3ab0b8" (UID: "18fec23f-6a90-42a0-8a23-f07eaf3ab0b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:22:18.349996 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:18.349978 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-istio-token" pod "18fec23f-6a90-42a0-8a23-f07eaf3ab0b8" (UID: "18fec23f-6a90-42a0-8a23-f07eaf3ab0b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:18.349996 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:18.349980 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-istio-envoy" pod "18fec23f-6a90-42a0-8a23-f07eaf3ab0b8" (UID: "18fec23f-6a90-42a0-8a23-f07eaf3ab0b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:18.448720 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:18.448684 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4gmc\" (UniqueName: \"kubernetes.io/projected/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-kube-api-access-m4gmc\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:18.448720 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:18.448717 4440 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-istio-token\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:18.448944 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:18.448732 4440 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-istio-podinfo\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:18.448944 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:18.448744 4440 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-credential-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:18.448944 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:18.448756 4440 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8-istio-envoy\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:19.156666 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:19.156627 4440 patch_prober.go:29] interesting pod/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:22:19.156862 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:19.156685 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" podUID="18fec23f-6a90-42a0-8a23-f07eaf3ab0b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:22:19.192754 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:19.192715 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" event={"ID":"18fec23f-6a90-42a0-8a23-f07eaf3ab0b8","Type":"ContainerDied","Data":"c114ca1e914d678bdb3ef2136a8a1eb825d017e535d6c7609f47f9b9fb62aac7"} Oct 31 01:22:19.192754 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:19.192743 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55" Oct 31 01:22:19.193233 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:19.192791 4440 scope.go:122] "RemoveContainer" containerID="5f844176526f4a4f9d20e6ba12035259140e24e5c1594b8d2e4ba28cb1cf5172" Oct 31 01:22:19.216575 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:19.216528 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55"] Oct 31 01:22:19.220126 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:19.220096 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2753ec-openshift-default-76f97dc989f7q55"] Oct 31 01:22:19.771362 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:19.771325 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18fec23f-6a90-42a0-8a23-f07eaf3ab0b8" path="/var/lib/kubelet/pods/18fec23f-6a90-42a0-8a23-f07eaf3ab0b8/volumes" Oct 31 01:22:23.028124 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:23.028077 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dql8q"] Oct 31 01:22:23.031433 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:23.031381 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dql8q"] Oct 31 01:22:23.770262 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:23.770227 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef1031f7-24de-4eb3-8d23-eb2db2be80fe" path="/var/lib/kubelet/pods/ef1031f7-24de-4eb3-8d23-eb2db2be80fe/volumes" Oct 31 01:22:25.082699 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:25.082663 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:22:29.316000 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:29.315960 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:22:31.001550 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:22:31.001509 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:22:32.908478 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:32.908431 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zgxj9"] Oct 31 01:22:32.908904 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:32.908662 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zgxj9" podUID="5c12334b-92c6-4d1c-b5a4-5ad834c842ff" containerName="manager" containerID="cri-o://5d844e8f5015cb9a4d93d3357384f45ff1fdece9bf65440cd36c8c0d424d5f3d" gracePeriod=10 Oct 31 01:22:33.182570 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:33.182535 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zgxj9" Oct 31 01:22:33.362310 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:33.362277 4440 generic.go:356] "Generic (PLEG): container finished" podID="5c12334b-92c6-4d1c-b5a4-5ad834c842ff" containerID="5d844e8f5015cb9a4d93d3357384f45ff1fdece9bf65440cd36c8c0d424d5f3d" exitCode=0 Oct 31 01:22:33.362519 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:33.362345 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zgxj9" event={"ID":"5c12334b-92c6-4d1c-b5a4-5ad834c842ff","Type":"ContainerDied","Data":"5d844e8f5015cb9a4d93d3357384f45ff1fdece9bf65440cd36c8c0d424d5f3d"} Oct 31 01:22:33.362519 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:33.362392 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zgxj9" Oct 31 01:22:33.362519 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:33.362444 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zgxj9" event={"ID":"5c12334b-92c6-4d1c-b5a4-5ad834c842ff","Type":"ContainerDied","Data":"286eb6702f6e69fbe15ff111228804d53ef8a22b7962f16a4325bc0a223561c8"} Oct 31 01:22:33.362519 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:33.362469 4440 scope.go:122] "RemoveContainer" containerID="5d844e8f5015cb9a4d93d3357384f45ff1fdece9bf65440cd36c8c0d424d5f3d" Oct 31 01:22:33.374328 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:33.374304 4440 scope.go:122] "RemoveContainer" containerID="5d844e8f5015cb9a4d93d3357384f45ff1fdece9bf65440cd36c8c0d424d5f3d" Oct 31 01:22:33.374757 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:22:33.374732 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d844e8f5015cb9a4d93d3357384f45ff1fdece9bf65440cd36c8c0d424d5f3d\": container with ID starting with 5d844e8f5015cb9a4d93d3357384f45ff1fdece9bf65440cd36c8c0d424d5f3d not found: ID does not exist" containerID="5d844e8f5015cb9a4d93d3357384f45ff1fdece9bf65440cd36c8c0d424d5f3d" Oct 31 01:22:33.374812 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:33.374767 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d844e8f5015cb9a4d93d3357384f45ff1fdece9bf65440cd36c8c0d424d5f3d"} err="failed to get container status \"5d844e8f5015cb9a4d93d3357384f45ff1fdece9bf65440cd36c8c0d424d5f3d\": rpc error: code = NotFound desc = could not find container \"5d844e8f5015cb9a4d93d3357384f45ff1fdece9bf65440cd36c8c0d424d5f3d\": container with ID starting with 5d844e8f5015cb9a4d93d3357384f45ff1fdece9bf65440cd36c8c0d424d5f3d not found: ID does not exist" Oct 31 01:22:33.376138 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:33.376115 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c12334b-92c6-4d1c-b5a4-5ad834c842ff-kube-api-access-l4424\" (UniqueName: \"kubernetes.io/projected/5c12334b-92c6-4d1c-b5a4-5ad834c842ff-kube-api-access-l4424\") pod \"5c12334b-92c6-4d1c-b5a4-5ad834c842ff\" (UID: \"5c12334b-92c6-4d1c-b5a4-5ad834c842ff\") " Oct 31 01:22:33.376243 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:33.376161 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c12334b-92c6-4d1c-b5a4-5ad834c842ff-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c12334b-92c6-4d1c-b5a4-5ad834c842ff-extensions-socket-volume\") pod \"5c12334b-92c6-4d1c-b5a4-5ad834c842ff\" (UID: \"5c12334b-92c6-4d1c-b5a4-5ad834c842ff\") " Oct 31 01:22:33.376562 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:33.376543 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c12334b-92c6-4d1c-b5a4-5ad834c842ff-extensions-socket-volume" pod "5c12334b-92c6-4d1c-b5a4-5ad834c842ff" (UID: "5c12334b-92c6-4d1c-b5a4-5ad834c842ff"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:33.378276 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:33.378253 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c12334b-92c6-4d1c-b5a4-5ad834c842ff-kube-api-access-l4424" pod "5c12334b-92c6-4d1c-b5a4-5ad834c842ff" (UID: "5c12334b-92c6-4d1c-b5a4-5ad834c842ff"). InnerVolumeSpecName "kube-api-access-l4424". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:33.477178 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:33.477142 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4424\" (UniqueName: \"kubernetes.io/projected/5c12334b-92c6-4d1c-b5a4-5ad834c842ff-kube-api-access-l4424\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:33.477178 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:33.477175 4440 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c12334b-92c6-4d1c-b5a4-5ad834c842ff-extensions-socket-volume\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:33.688538 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:33.688500 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zgxj9"] Oct 31 01:22:33.692985 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:33.692953 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zgxj9"] Oct 31 01:22:33.770277 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:33.770193 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c12334b-92c6-4d1c-b5a4-5ad834c842ff" path="/var/lib/kubelet/pods/5c12334b-92c6-4d1c-b5a4-5ad834c842ff/volumes" Oct 31 01:22:36.685797 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:36.685757 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:37.196175 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:37.196139 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:42.721903 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:42.721858 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:43.597668 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:43.597606 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:43.700619 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:43.700583 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:47.056780 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:47.056740 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:47.162320 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:47.162279 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:49.022485 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:49.022413 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dbvcx"] Oct 31 01:22:49.025322 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:49.025286 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dbvcx"] Oct 31 01:22:49.773589 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:49.773542 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33be65c2-e4d4-4fd8-bf80-1c3d9be57bee" path="/var/lib/kubelet/pods/33be65c2-e4d4-4fd8-bf80-1c3d9be57bee/volumes" Oct 31 01:22:56.711946 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:56.711907 4440 scope.go:122] "RemoveContainer" containerID="551aaeae72aa26c6ff19155a9eaafae0c75e2c4c4e87df0feaea40af01c3ff94" Oct 31 01:22:56.721393 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:22:56.721372 4440 scope.go:122] "RemoveContainer" containerID="d7193ceae87970ebee59011b63672f948579f177638e4272b820d7db25b0054c" Oct 31 01:23:00.023256 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:00.023214 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mxkq4"] Oct 31 01:23:00.032342 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:00.028877 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mxkq4"] Oct 31 01:23:01.017235 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:23:01.017185 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:23:01.769773 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:01.769736 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e76c45a-690e-4796-ae06-6c5147d9990e" path="/var/lib/kubelet/pods/7e76c45a-690e-4796-ae06-6c5147d9990e/volumes" Oct 31 01:23:19.202226 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:19.202187 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:19.309922 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:19.309881 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:20.022844 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:20.022797 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xqs8b"] Oct 31 01:23:20.025665 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:20.025637 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-xqs8b"] Oct 31 01:23:21.770305 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:21.770268 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d32e347-1610-48e9-bcbb-25005fc2e92a" path="/var/lib/kubelet/pods/4d32e347-1610-48e9-bcbb-25005fc2e92a/volumes" Oct 31 01:23:31.000887 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:23:31.000846 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:23:33.947672 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:33.947637 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2j5x4"] Oct 31 01:23:33.948148 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:33.948045 4440 cpu_manager.go:414] "containerMap: removing container" podUID="4c4c9d10-144f-4e32-8f17-45bea8e215a3" containerName="maas-api" Oct 31 01:23:33.948148 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:33.948056 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c4c9d10-144f-4e32-8f17-45bea8e215a3" containerName="maas-api" Oct 31 01:23:33.948148 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:33.948074 4440 cpu_manager.go:414] "containerMap: removing container" podUID="5c12334b-92c6-4d1c-b5a4-5ad834c842ff" containerName="manager" Oct 31 01:23:33.948148 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:33.948078 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c12334b-92c6-4d1c-b5a4-5ad834c842ff" containerName="manager" Oct 31 01:23:33.948148 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:33.948095 4440 cpu_manager.go:414] "containerMap: removing container" podUID="18fec23f-6a90-42a0-8a23-f07eaf3ab0b8" containerName="istio-proxy" Oct 31 01:23:33.948148 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:33.948099 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18fec23f-6a90-42a0-8a23-f07eaf3ab0b8" containerName="istio-proxy" Oct 31 01:23:33.948148 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:33.948113 4440 cpu_manager.go:414] "containerMap: removing container" podUID="cc93cb50-1b43-46a3-940a-f3d302743d4d" containerName="main" Oct 31 01:23:33.948148 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:33.948118 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc93cb50-1b43-46a3-940a-f3d302743d4d" containerName="main" Oct 31 01:23:33.948148 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:33.948126 4440 cpu_manager.go:414] "containerMap: removing container" podUID="18fec23f-6a90-42a0-8a23-f07eaf3ab0b8" containerName="istio-proxy" Oct 31 01:23:33.948148 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:33.948131 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18fec23f-6a90-42a0-8a23-f07eaf3ab0b8" containerName="istio-proxy" Oct 31 01:23:33.948540 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:33.948188 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c12334b-92c6-4d1c-b5a4-5ad834c842ff" containerName="manager" Oct 31 01:23:33.948540 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:33.948197 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="18fec23f-6a90-42a0-8a23-f07eaf3ab0b8" containerName="istio-proxy" Oct 31 01:23:33.948540 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:33.948205 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc93cb50-1b43-46a3-940a-f3d302743d4d" containerName="main" Oct 31 01:23:33.948540 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:33.948210 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="18fec23f-6a90-42a0-8a23-f07eaf3ab0b8" containerName="istio-proxy" Oct 31 01:23:33.948540 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:33.948218 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c4c9d10-144f-4e32-8f17-45bea8e215a3" containerName="maas-api" Oct 31 01:23:33.951670 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:33.951648 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2j5x4" Oct 31 01:23:33.955983 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:33.955956 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-cblcn\"" Oct 31 01:23:33.964524 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:33.964495 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2j5x4"] Oct 31 01:23:34.070526 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:34.070477 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x5sl\" (UniqueName: \"kubernetes.io/projected/189472bb-44c7-4a81-bff1-49b621da9b02-kube-api-access-5x5sl\") pod \"kuadrant-operator-controller-manager-678d87766f-2j5x4\" (UID: \"189472bb-44c7-4a81-bff1-49b621da9b02\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2j5x4" Oct 31 01:23:34.070753 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:34.070550 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/189472bb-44c7-4a81-bff1-49b621da9b02-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-2j5x4\" (UID: \"189472bb-44c7-4a81-bff1-49b621da9b02\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2j5x4" Oct 31 01:23:34.171841 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:34.171798 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x5sl\" (UniqueName: \"kubernetes.io/projected/189472bb-44c7-4a81-bff1-49b621da9b02-kube-api-access-5x5sl\") pod \"kuadrant-operator-controller-manager-678d87766f-2j5x4\" (UID: \"189472bb-44c7-4a81-bff1-49b621da9b02\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2j5x4" Oct 31 01:23:34.172048 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:34.171890 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/189472bb-44c7-4a81-bff1-49b621da9b02-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-2j5x4\" (UID: \"189472bb-44c7-4a81-bff1-49b621da9b02\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2j5x4" Oct 31 01:23:34.172300 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:34.172281 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/189472bb-44c7-4a81-bff1-49b621da9b02-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-2j5x4\" (UID: \"189472bb-44c7-4a81-bff1-49b621da9b02\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2j5x4" Oct 31 01:23:34.180961 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:34.180937 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x5sl\" (UniqueName: \"kubernetes.io/projected/189472bb-44c7-4a81-bff1-49b621da9b02-kube-api-access-5x5sl\") pod \"kuadrant-operator-controller-manager-678d87766f-2j5x4\" (UID: \"189472bb-44c7-4a81-bff1-49b621da9b02\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2j5x4" Oct 31 01:23:34.267747 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:34.267652 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2j5x4" Oct 31 01:23:34.553470 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:34.553435 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2j5x4"] Oct 31 01:23:34.554380 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:23:34.554347 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod189472bb_44c7_4a81_bff1_49b621da9b02.slice/crio-9a7e36d49f4ad0c8bfb517b01c8ab89b5ff1d4011f80e616489bd8571a0da10a WatchSource:0}: Error finding container 9a7e36d49f4ad0c8bfb517b01c8ab89b5ff1d4011f80e616489bd8571a0da10a: Status 404 returned error can't find the container with id 9a7e36d49f4ad0c8bfb517b01c8ab89b5ff1d4011f80e616489bd8571a0da10a Oct 31 01:23:34.746803 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:34.746769 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2j5x4" event={"ID":"189472bb-44c7-4a81-bff1-49b621da9b02","Type":"ContainerStarted","Data":"dfae59c0f40730ae2b84b8d94bc5a29c353f715fef2aa4686817c735c4b3d142"} Oct 31 01:23:34.747025 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:34.746824 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2j5x4" event={"ID":"189472bb-44c7-4a81-bff1-49b621da9b02","Type":"ContainerStarted","Data":"9a7e36d49f4ad0c8bfb517b01c8ab89b5ff1d4011f80e616489bd8571a0da10a"} Oct 31 01:23:34.747025 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:34.746862 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2j5x4" Oct 31 01:23:34.774510 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:34.774449 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2j5x4" podStartSLOduration=1.774430509 podStartE2EDuration="1.774430509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:23:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:23:34.773248019 +0000 UTC m=+1659.131970332" watchObservedRunningTime="2026-10-31 01:23:34.774430509 +0000 UTC m=+1659.133152822" Oct 31 01:23:39.023349 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:39.023250 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5kf2p"] Oct 31 01:23:39.026069 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:39.026040 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5kf2p"] Oct 31 01:23:39.770508 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:39.770467 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae2356b6-2a2e-48b2-b1a4-4856586c3e36" path="/var/lib/kubelet/pods/ae2356b6-2a2e-48b2-b1a4-4856586c3e36/volumes" Oct 31 01:23:45.755448 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:45.755389 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2j5x4" Oct 31 01:23:47.124317 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:47.124275 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:47.244384 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:47.244350 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:53.413069 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:53.413025 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:23:56.770655 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:56.770534 4440 scope.go:122] "RemoveContainer" containerID="e4628487e59a5b221c4cacdc3962848d57b68e1e407927dfa7ed41fe88e55768" Oct 31 01:23:56.806576 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:56.806530 4440 scope.go:122] "RemoveContainer" containerID="15c996cea89ee36d41cfcf1d71c01a2487086e21060b48686650033646cc140f" Oct 31 01:23:56.821293 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:56.821268 4440 scope.go:122] "RemoveContainer" containerID="28ac6ddb0c8dddb7ea6999f15568465e1bfb9d59cee83572ff2dfa1cef6baca6" Oct 31 01:23:56.837139 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:56.836619 4440 scope.go:122] "RemoveContainer" containerID="79a9578a68d2ccce4914caa27c4a514447d1360c248b54c794e7f4bdc8a9e538" Oct 31 01:23:57.026221 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:57.026135 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hbhgd"] Oct 31 01:23:57.030261 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:57.030234 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hbhgd"] Oct 31 01:23:57.770911 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:23:57.770875 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33492813-313f-430e-bf1f-6676b16e42b8" path="/var/lib/kubelet/pods/33492813-313f-430e-bf1f-6676b16e42b8/volumes" Oct 31 01:24:01.019684 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:24:01.019646 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:24:08.459375 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:08.459331 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:24:12.621018 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:12.620959 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:24:15.029382 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:15.029340 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k84wf"] Oct 31 01:24:15.032229 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:15.032194 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k84wf"] Oct 31 01:24:15.776001 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:15.775965 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc5d14b8-cd36-4d76-a8f7-4e000c6238ec" path="/var/lib/kubelet/pods/fc5d14b8-cd36-4d76-a8f7-4e000c6238ec/volumes" Oct 31 01:24:27.258896 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:27.258849 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:24:28.168768 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:28.168726 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:28.564109 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:28.564015 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz"] Oct 31 01:24:28.568059 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:28.568029 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" Oct 31 01:24:28.572344 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:28.572319 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-b93c48-openshift-default-dockercfg-nltrs\"" Oct 31 01:24:28.585790 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:28.585761 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz"] Oct 31 01:24:28.603694 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:28.603654 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-workload-certs\") pod \"e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz\" (UID: \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\") " pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" Oct 31 01:24:28.604021 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:28.603711 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-istio-token\") pod \"e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz\" (UID: \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\") " pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" Oct 31 01:24:28.604021 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:28.603838 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-workload-socket\") pod \"e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz\" (UID: \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\") " pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" Oct 31 01:24:28.604021 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:28.603899 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-credential-socket\") pod \"e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz\" (UID: \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\") " pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" Oct 31 01:24:28.604237 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:28.604011 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-istio-podinfo\") pod \"e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz\" (UID: \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\") " pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" Oct 31 01:24:28.604237 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:28.604051 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xqw2\" (UniqueName: \"kubernetes.io/projected/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-kube-api-access-2xqw2\") pod \"e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz\" (UID: \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\") " pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" Oct 31 01:24:28.604237 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:28.604152 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-istio-envoy\") pod \"e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz\" (UID: \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\") " pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" Oct 31 01:24:28.604237 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:28.604218 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-istio-data\") pod \"e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz\" (UID: \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\") " pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" Oct 31 01:24:28.604561 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:28.604256 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-istiod-ca-cert\") pod \"e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz\" (UID: \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\") " pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" Oct 31 01:24:28.631105 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:28.631061 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:28.705888 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:28.705849 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-istio-podinfo\") pod \"e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz\" (UID: \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\") " pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" Oct 31 01:24:28.705888 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:28.705892 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xqw2\" (UniqueName: \"kubernetes.io/projected/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-kube-api-access-2xqw2\") pod \"e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz\" (UID: \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\") " pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" Oct 31 01:24:28.706173 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:28.705926 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-istio-envoy\") pod \"e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz\" (UID: \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\") " pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" Oct 31 01:24:28.706173 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:28.705973 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-istio-data\") pod \"e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz\" (UID: \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\") " pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" Oct 31 01:24:28.706173 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:28.706006 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-istiod-ca-cert\") pod \"e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz\" (UID: \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\") " pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" Oct 31 01:24:28.706173 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:28.706034 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-workload-certs\") pod \"e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz\" (UID: \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\") " pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" Oct 31 01:24:28.706173 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:28.706071 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-istio-token\") pod \"e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz\" (UID: \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\") " pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" Oct 31 01:24:28.706173 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:28.706131 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-workload-socket\") pod \"e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz\" (UID: \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\") " pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" Oct 31 01:24:28.706511 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:28.706207 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-credential-socket\") pod \"e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz\" (UID: \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\") " pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" Oct 31 01:24:28.706702 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:28.706653 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-credential-socket\") pod \"e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz\" (UID: \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\") " pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" Oct 31 01:24:28.707050 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:28.707025 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-workload-certs\") pod \"e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz\" (UID: \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\") " pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" Oct 31 01:24:28.707116 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:28.707075 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-istiod-ca-cert\") pod \"e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz\" (UID: \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\") " pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" Oct 31 01:24:28.707817 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:28.707784 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-istio-data\") pod \"e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz\" (UID: \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\") " pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" Oct 31 01:24:28.707941 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:28.707835 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-workload-socket\") pod \"e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz\" (UID: \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\") " pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" Oct 31 01:24:28.709782 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:28.709752 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-istio-podinfo\") pod \"e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz\" (UID: \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\") " pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" Oct 31 01:24:28.717310 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:28.716572 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-istio-token\") pod \"e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz\" (UID: \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\") " pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" Oct 31 01:24:28.717310 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:28.717230 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-istio-envoy\") pod \"e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz\" (UID: \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\") " pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" Oct 31 01:24:28.720147 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:28.720115 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xqw2\" (UniqueName: \"kubernetes.io/projected/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-kube-api-access-2xqw2\") pod \"e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz\" (UID: \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\") " pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" Oct 31 01:24:28.768503 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:28.768455 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:28.884339 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:28.884250 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" Oct 31 01:24:29.170729 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:29.170698 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz"] Oct 31 01:24:29.172547 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:24:29.172518 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ccb6a7f_bc35_4a1b_b40d_8255cd731f19.slice/crio-b4c7dce401d399c3ae2794c77a2566c5095767cd585ac735531825e9b291f51a WatchSource:0}: Error finding container b4c7dce401d399c3ae2794c77a2566c5095767cd585ac735531825e9b291f51a: Status 404 returned error can't find the container with id b4c7dce401d399c3ae2794c77a2566c5095767cd585ac735531825e9b291f51a Oct 31 01:24:29.174755 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:29.174722 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:24:29.174857 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:29.174790 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:24:29.174857 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:29.174828 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:24:29.196194 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:29.195847 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" event={"ID":"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19","Type":"ContainerStarted","Data":"b4c7dce401d399c3ae2794c77a2566c5095767cd585ac735531825e9b291f51a"} Oct 31 01:24:30.204941 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:30.204899 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" event={"ID":"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19","Type":"ContainerStarted","Data":"735d187006bb576f1734d44a6d73fb00e20d658165e52fbdb76d047637aecc87"} Oct 31 01:24:30.226960 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:30.226905 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" podStartSLOduration=2.226892396 podStartE2EDuration="2.226892396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:30.225067734 +0000 UTC m=+1714.583790049" watchObservedRunningTime="2026-10-31 01:24:30.226892396 +0000 UTC m=+1714.585614712" Oct 31 01:24:30.884520 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:30.884477 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" Oct 31 01:24:30.889946 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:30.889917 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" Oct 31 01:24:31.002798 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:24:31.002758 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:24:31.212090 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:31.212062 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" Oct 31 01:24:31.213177 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:31.213155 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" Oct 31 01:24:32.450683 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:32.450645 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:32.569450 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:32.569390 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:33.619407 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:33.619365 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b93c48-77dc6785cc-t8dxx"] Oct 31 01:24:33.623610 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:33.623584 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b93c48-77dc6785cc-t8dxx" Oct 31 01:24:33.636063 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:33.636029 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:24:33.636234 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:33.636142 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-n5p2t\"" Oct 31 01:24:33.636234 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:33.636203 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-b93c48-serving-cert\"" Oct 31 01:24:33.636319 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:33.636276 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:24:33.637057 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:33.637036 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 01:24:33.647704 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:33.647672 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b93c48-77dc6785cc-t8dxx"] Oct 31 01:24:33.676672 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:33.676628 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:33.761600 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:33.761547 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eafc1139-7ac3-44b9-8198-2bf94485dc75-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b93c48-77dc6785cc-t8dxx\" (UID: \"eafc1139-7ac3-44b9-8198-2bf94485dc75\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b93c48-77dc6785cc-t8dxx" Oct 31 01:24:33.761824 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:33.761774 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eafc1139-7ac3-44b9-8198-2bf94485dc75-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b93c48-77dc6785cc-t8dxx\" (UID: \"eafc1139-7ac3-44b9-8198-2bf94485dc75\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b93c48-77dc6785cc-t8dxx" Oct 31 01:24:33.761824 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:33.761811 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bw9r\" (UniqueName: \"kubernetes.io/projected/eafc1139-7ac3-44b9-8198-2bf94485dc75-kube-api-access-8bw9r\") pod \"maas-api-e2e-worker-single-main-b93c48-77dc6785cc-t8dxx\" (UID: \"eafc1139-7ac3-44b9-8198-2bf94485dc75\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b93c48-77dc6785cc-t8dxx" Oct 31 01:24:33.761992 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:33.761970 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eafc1139-7ac3-44b9-8198-2bf94485dc75-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b93c48-77dc6785cc-t8dxx\" (UID: \"eafc1139-7ac3-44b9-8198-2bf94485dc75\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b93c48-77dc6785cc-t8dxx" Oct 31 01:24:33.804406 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:33.804363 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:33.863650 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:33.863613 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eafc1139-7ac3-44b9-8198-2bf94485dc75-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b93c48-77dc6785cc-t8dxx\" (UID: \"eafc1139-7ac3-44b9-8198-2bf94485dc75\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b93c48-77dc6785cc-t8dxx" Oct 31 01:24:33.863873 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:33.863654 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bw9r\" (UniqueName: \"kubernetes.io/projected/eafc1139-7ac3-44b9-8198-2bf94485dc75-kube-api-access-8bw9r\") pod \"maas-api-e2e-worker-single-main-b93c48-77dc6785cc-t8dxx\" (UID: \"eafc1139-7ac3-44b9-8198-2bf94485dc75\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b93c48-77dc6785cc-t8dxx" Oct 31 01:24:33.863873 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:33.863707 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eafc1139-7ac3-44b9-8198-2bf94485dc75-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b93c48-77dc6785cc-t8dxx\" (UID: \"eafc1139-7ac3-44b9-8198-2bf94485dc75\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b93c48-77dc6785cc-t8dxx" Oct 31 01:24:33.863873 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:33.863736 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eafc1139-7ac3-44b9-8198-2bf94485dc75-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b93c48-77dc6785cc-t8dxx\" (UID: \"eafc1139-7ac3-44b9-8198-2bf94485dc75\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b93c48-77dc6785cc-t8dxx" Oct 31 01:24:33.864527 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:33.864497 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eafc1139-7ac3-44b9-8198-2bf94485dc75-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b93c48-77dc6785cc-t8dxx\" (UID: \"eafc1139-7ac3-44b9-8198-2bf94485dc75\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b93c48-77dc6785cc-t8dxx" Oct 31 01:24:33.867146 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:33.867096 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eafc1139-7ac3-44b9-8198-2bf94485dc75-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b93c48-77dc6785cc-t8dxx\" (UID: \"eafc1139-7ac3-44b9-8198-2bf94485dc75\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b93c48-77dc6785cc-t8dxx" Oct 31 01:24:33.867533 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:33.867512 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eafc1139-7ac3-44b9-8198-2bf94485dc75-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b93c48-77dc6785cc-t8dxx\" (UID: \"eafc1139-7ac3-44b9-8198-2bf94485dc75\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b93c48-77dc6785cc-t8dxx" Oct 31 01:24:33.876313 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:33.876251 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bw9r\" (UniqueName: \"kubernetes.io/projected/eafc1139-7ac3-44b9-8198-2bf94485dc75-kube-api-access-8bw9r\") pod \"maas-api-e2e-worker-single-main-b93c48-77dc6785cc-t8dxx\" (UID: \"eafc1139-7ac3-44b9-8198-2bf94485dc75\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b93c48-77dc6785cc-t8dxx" Oct 31 01:24:33.940925 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:33.940892 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b93c48-77dc6785cc-t8dxx" Oct 31 01:24:34.005617 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:34.005577 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:34.227923 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:34.227894 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b93c48-77dc6785cc-t8dxx"] Oct 31 01:24:34.229267 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:24:34.229232 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeafc1139_7ac3_44b9_8198_2bf94485dc75.slice/crio-746582e19d8a159509c30ea5b48e66b12c3a47fece1538e904d2643b59e636b4 WatchSource:0}: Error finding container 746582e19d8a159509c30ea5b48e66b12c3a47fece1538e904d2643b59e636b4: Status 404 returned error can't find the container with id 746582e19d8a159509c30ea5b48e66b12c3a47fece1538e904d2643b59e636b4 Oct 31 01:24:34.511513 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:34.511397 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:35.205694 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:35.205658 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:35.247273 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:35.247235 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b93c48-77dc6785cc-t8dxx" event={"ID":"eafc1139-7ac3-44b9-8198-2bf94485dc75","Type":"ContainerStarted","Data":"ea4d23e7c952840eef751ff5ce04995b1c00bbb2d0fe6e2ed618159eeee345d4"} Oct 31 01:24:35.247450 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:35.247299 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b93c48-77dc6785cc-t8dxx" event={"ID":"eafc1139-7ac3-44b9-8198-2bf94485dc75","Type":"ContainerStarted","Data":"746582e19d8a159509c30ea5b48e66b12c3a47fece1538e904d2643b59e636b4"} Oct 31 01:24:35.247450 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:35.247409 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b93c48-77dc6785cc-t8dxx" Oct 31 01:24:35.267907 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:35.267853 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b93c48-77dc6785cc-t8dxx" podStartSLOduration=1.787209697 podStartE2EDuration="2.267838437s" totalImagesPullingTime="480.62874ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:35.265221697 +0000 UTC m=+1719.623944012" watchObservedRunningTime="2026-10-31 01:24:35.267838437 +0000 UTC m=+1719.626560753" Oct 31 01:24:35.810405 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:35.810366 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:24:35.984767 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:35.984728 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:36.923129 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:36.923084 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:37.710927 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:37.710889 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:38.461326 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:38.461287 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:39.085165 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:39.085133 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:39.721440 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:39.721177 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:40.278021 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:40.277990 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:40.889871 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:40.889833 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:41.259798 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:41.259769 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b93c48-77dc6785cc-t8dxx" Oct 31 01:24:41.482999 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:41.482952 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:42.168394 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:42.168354 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:42.834856 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:42.834813 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:43.631429 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:43.631381 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:44.195544 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:44.195504 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:44.889661 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:44.889624 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:45.311610 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:45.311528 4440 generic.go:356] "Generic (PLEG): container finished" podID="2ccb6a7f-bc35-4a1b-b40d-8255cd731f19" containerID="735d187006bb576f1734d44a6d73fb00e20d658165e52fbdb76d047637aecc87" exitCode=0 Oct 31 01:24:45.311813 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:45.311602 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" event={"ID":"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19","Type":"ContainerDied","Data":"735d187006bb576f1734d44a6d73fb00e20d658165e52fbdb76d047637aecc87"} Oct 31 01:24:45.312009 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:45.311994 4440 scope.go:122] "RemoveContainer" containerID="735d187006bb576f1734d44a6d73fb00e20d658165e52fbdb76d047637aecc87" Oct 31 01:24:45.313268 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:45.313243 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:24:45.313328 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:45.313307 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:24:45.313364 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:45.313340 4440 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 01:24:45.617281 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:45.617232 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:46.214447 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:46.213528 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" Oct 31 01:24:46.283936 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:46.283895 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:46.343693 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:46.343656 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" event={"ID":"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19","Type":"ContainerStarted","Data":"3d505a1c1bfa3bb5f9d4e3f595f59c63b0819eac5124d5a55ede0d9b67e8fb00"} Oct 31 01:24:46.884645 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:46.884609 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" Oct 31 01:24:46.885893 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:46.885866 4440 patch_prober.go:29] interesting pod/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 01:24:46.886008 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:46.885919 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" podUID="2ccb6a7f-bc35-4a1b-b40d-8255cd731f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 01:24:46.918742 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:46.918688 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:47.132476 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:47.132410 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:47.462205 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:47.462155 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:47.595483 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:47.595409 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:47.885014 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:47.884930 4440 patch_prober.go:29] interesting pod/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 01:24:47.885014 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:47.884984 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" podUID="2ccb6a7f-bc35-4a1b-b40d-8255cd731f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 01:24:48.089919 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:48.089876 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:48.219443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:48.219049 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:48.354510 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:48.354473 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:48.884644 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:48.884612 4440 patch_prober.go:29] interesting pod/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 01:24:48.885047 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:48.884663 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" podUID="2ccb6a7f-bc35-4a1b-b40d-8255cd731f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 01:24:49.078880 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:49.078840 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-b93c48/simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg"] Oct 31 01:24:49.082789 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:49.082762 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b93c48/simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg" Oct 31 01:24:49.125217 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:49.124962 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-b93c48\"/\"simulator-single-main-b93c48-kserve-self-signed-certs\"" Oct 31 01:24:49.125217 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:49.125106 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-b93c48\"/\"default-dockercfg-k5s4f\"" Oct 31 01:24:49.127145 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:49.127110 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-b93c48/simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg"] Oct 31 01:24:49.142523 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:49.142440 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-b93c48\"/\"kube-root-ca.crt\"" Oct 31 01:24:49.142697 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:49.142533 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-b93c48\"/\"openshift-service-ca.crt\"" Oct 31 01:24:49.165801 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:49.165765 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635-home\") pod \"simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg\" (UID: \"ecca2fe3-c9c5-4d53-a4a1-96d9dce86635\") " pod="e2e-models-e2e-worker-single-main-b93c48/simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg" Oct 31 01:24:49.165801 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:49.165803 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635-tls-certs\") pod \"simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg\" (UID: \"ecca2fe3-c9c5-4d53-a4a1-96d9dce86635\") " pod="e2e-models-e2e-worker-single-main-b93c48/simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg" Oct 31 01:24:49.166067 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:49.165959 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635-model-cache\") pod \"simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg\" (UID: \"ecca2fe3-c9c5-4d53-a4a1-96d9dce86635\") " pod="e2e-models-e2e-worker-single-main-b93c48/simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg" Oct 31 01:24:49.166067 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:49.166015 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq966\" (UniqueName: \"kubernetes.io/projected/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635-kube-api-access-nq966\") pod \"simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg\" (UID: \"ecca2fe3-c9c5-4d53-a4a1-96d9dce86635\") " pod="e2e-models-e2e-worker-single-main-b93c48/simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg" Oct 31 01:24:49.166183 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:49.166167 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635-tmp-dir\") pod \"simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg\" (UID: \"ecca2fe3-c9c5-4d53-a4a1-96d9dce86635\") " pod="e2e-models-e2e-worker-single-main-b93c48/simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg" Oct 31 01:24:49.166244 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:49.166204 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635-dshm\") pod \"simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg\" (UID: \"ecca2fe3-c9c5-4d53-a4a1-96d9dce86635\") " pod="e2e-models-e2e-worker-single-main-b93c48/simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg" Oct 31 01:24:49.170087 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:49.170061 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:49.267869 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:49.267822 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq966\" (UniqueName: \"kubernetes.io/projected/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635-kube-api-access-nq966\") pod \"simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg\" (UID: \"ecca2fe3-c9c5-4d53-a4a1-96d9dce86635\") " pod="e2e-models-e2e-worker-single-main-b93c48/simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg" Oct 31 01:24:49.268618 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:49.268589 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635-tmp-dir\") pod \"simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg\" (UID: \"ecca2fe3-c9c5-4d53-a4a1-96d9dce86635\") " pod="e2e-models-e2e-worker-single-main-b93c48/simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg" Oct 31 01:24:49.269305 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:49.269279 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635-dshm\") pod \"simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg\" (UID: \"ecca2fe3-c9c5-4d53-a4a1-96d9dce86635\") " pod="e2e-models-e2e-worker-single-main-b93c48/simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg" Oct 31 01:24:49.269533 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:49.269332 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635-home\") pod \"simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg\" (UID: \"ecca2fe3-c9c5-4d53-a4a1-96d9dce86635\") " pod="e2e-models-e2e-worker-single-main-b93c48/simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg" Oct 31 01:24:49.269533 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:49.269364 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635-tls-certs\") pod \"simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg\" (UID: \"ecca2fe3-c9c5-4d53-a4a1-96d9dce86635\") " pod="e2e-models-e2e-worker-single-main-b93c48/simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg" Oct 31 01:24:49.269533 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:49.269223 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635-tmp-dir\") pod \"simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg\" (UID: \"ecca2fe3-c9c5-4d53-a4a1-96d9dce86635\") " pod="e2e-models-e2e-worker-single-main-b93c48/simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg" Oct 31 01:24:49.270041 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:49.270016 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635-model-cache\") pod \"simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg\" (UID: \"ecca2fe3-c9c5-4d53-a4a1-96d9dce86635\") " pod="e2e-models-e2e-worker-single-main-b93c48/simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg" Oct 31 01:24:49.270351 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:49.270330 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635-model-cache\") pod \"simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg\" (UID: \"ecca2fe3-c9c5-4d53-a4a1-96d9dce86635\") " pod="e2e-models-e2e-worker-single-main-b93c48/simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg" Oct 31 01:24:49.270948 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:49.270921 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635-home\") pod \"simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg\" (UID: \"ecca2fe3-c9c5-4d53-a4a1-96d9dce86635\") " pod="e2e-models-e2e-worker-single-main-b93c48/simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg" Oct 31 01:24:49.273925 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:49.273735 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635-dshm\") pod \"simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg\" (UID: \"ecca2fe3-c9c5-4d53-a4a1-96d9dce86635\") " pod="e2e-models-e2e-worker-single-main-b93c48/simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg" Oct 31 01:24:49.275723 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:49.275675 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635-tls-certs\") pod \"simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg\" (UID: \"ecca2fe3-c9c5-4d53-a4a1-96d9dce86635\") " pod="e2e-models-e2e-worker-single-main-b93c48/simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg" Oct 31 01:24:49.282875 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:49.282844 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq966\" (UniqueName: \"kubernetes.io/projected/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635-kube-api-access-nq966\") pod \"simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg\" (UID: \"ecca2fe3-c9c5-4d53-a4a1-96d9dce86635\") " pod="e2e-models-e2e-worker-single-main-b93c48/simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg" Oct 31 01:24:49.367028 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:49.366978 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:49.395114 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:49.395014 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b93c48/simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg" Oct 31 01:24:49.713646 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:49.713600 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-b93c48/simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg"] Oct 31 01:24:49.730380 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:24:49.730350 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecca2fe3_c9c5_4d53_a4a1_96d9dce86635.slice/crio-d95545164d962c5e886264794c8c9c2b49d08dda3763109a3273790b7bd1f75f WatchSource:0}: Error finding container d95545164d962c5e886264794c8c9c2b49d08dda3763109a3273790b7bd1f75f: Status 404 returned error can't find the container with id d95545164d962c5e886264794c8c9c2b49d08dda3763109a3273790b7bd1f75f Oct 31 01:24:49.885669 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:49.885630 4440 patch_prober.go:29] interesting pod/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 01:24:49.886255 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:49.886216 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" podUID="2ccb6a7f-bc35-4a1b-b40d-8255cd731f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 01:24:49.922444 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:49.915497 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:50.371394 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:50.371348 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b93c48/simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg" event={"ID":"ecca2fe3-c9c5-4d53-a4a1-96d9dce86635","Type":"ContainerStarted","Data":"3fc6ae618752585150302663816b1babb114715d764d36e555b2b38efd0344f2"} Oct 31 01:24:50.371597 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:50.371411 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b93c48/simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg" event={"ID":"ecca2fe3-c9c5-4d53-a4a1-96d9dce86635","Type":"ContainerStarted","Data":"d95545164d962c5e886264794c8c9c2b49d08dda3763109a3273790b7bd1f75f"} Oct 31 01:24:50.388525 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:50.388462 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-b93c48/simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg" podStartSLOduration=1.388444677 podStartE2EDuration="1.388444677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:50.387465989 +0000 UTC m=+1734.746188306" watchObservedRunningTime="2026-10-31 01:24:50.388444677 +0000 UTC m=+1734.747167116" Oct 31 01:24:50.395125 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:50.395095 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-b93c48/simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg" Oct 31 01:24:50.559558 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:50.559519 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:50.884750 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:50.884715 4440 patch_prober.go:29] interesting pod/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 01:24:50.884923 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:50.884769 4440 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" podUID="2ccb6a7f-bc35-4a1b-b40d-8255cd731f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 01:24:51.123615 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:51.123579 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:24:51.366108 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:51.366070 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:51.928922 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:51.928883 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" Oct 31 01:24:51.929337 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:51.929324 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" Oct 31 01:24:51.930153 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:51.930130 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" Oct 31 01:24:52.195601 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:52.195505 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:52.856990 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:52.856955 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:53.436796 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:53.436751 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:54.104400 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:54.104353 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:54.816841 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:54.816792 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:55.405257 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:55.405220 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:56.048845 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:56.048805 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:56.693203 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:56.693165 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:56.893164 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:56.893128 4440 scope.go:122] "RemoveContainer" containerID="9b7dc877756bd44ef405821e4358cf77d1150c395003fdb2c105e2f4b0ee4e0c" Oct 31 01:24:56.903346 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:56.903321 4440 scope.go:122] "RemoveContainer" containerID="9ebe7641ec200120fd7eb5586317aafd63776719b1af84b52aed61459691b1af" Oct 31 01:24:56.912383 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:56.912347 4440 scope.go:122] "RemoveContainer" containerID="8aa297936514083962b0472441ea159c4a27707240c45e2f5d323900f1b95e9e" Oct 31 01:24:57.284803 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:57.284760 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:57.887303 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:57.887261 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:58.559148 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:58.559106 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:59.190775 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:59.190727 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:59.395796 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:59.395743 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-b93c48/simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg" Oct 31 01:24:59.409001 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:59.408968 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-b93c48/simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg" Oct 31 01:24:59.463485 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:59.463379 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-b93c48/simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg" Oct 31 01:24:59.999373 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:24:59.999341 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:00.693853 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:00.693818 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:01.001396 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:25:01.001297 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:25:01.362004 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:01.361914 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:01.999858 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:01.999817 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:02.693743 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:02.693709 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:03.303913 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:03.303872 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:04.012997 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:04.012953 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:04.660532 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:04.660492 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:05.297799 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:05.297759 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:05.993913 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:05.993871 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:06.655780 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:06.655740 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:07.282954 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:07.282917 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:07.378147 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:07.378109 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:25:08.095095 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:08.095058 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:08.786719 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:08.786667 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:09.399133 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:09.399092 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:10.043019 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:10.042969 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:10.681023 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:10.680981 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:11.564709 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:11.564676 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:12.021498 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:12.021458 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-t29mq"] Oct 31 01:25:12.027074 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:12.027040 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-t29mq"] Oct 31 01:25:12.311294 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:12.311188 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:13.771168 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:13.771131 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e63f5103-aba8-4617-b467-71bf70b5d456" path="/var/lib/kubelet/pods/e63f5103-aba8-4617-b467-71bf70b5d456/volumes" Oct 31 01:25:13.975336 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:13.975293 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:14.197558 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:14.197516 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:15.006605 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:15.006562 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:15.543267 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:15.543231 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:15.684099 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:15.684055 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:16.298943 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:16.298903 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:17.024994 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:17.024942 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:17.696623 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:17.696579 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:18.378566 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:18.378528 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:18.997522 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:18.997474 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:19.603358 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:19.603315 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:20.299819 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:20.299786 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:20.918303 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:20.918263 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:21.688394 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:21.688352 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:22.298620 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:22.298577 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:22.896376 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:22.896333 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:23.024523 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:23.024479 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xtkrd"] Oct 31 01:25:23.029680 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:23.029642 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xtkrd"] Oct 31 01:25:23.662114 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:23.662060 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:23.770562 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:23.770518 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7e16c3b-a22a-4460-8499-e17f49ce062f" path="/var/lib/kubelet/pods/b7e16c3b-a22a-4460-8499-e17f49ce062f/volumes" Oct 31 01:25:24.216059 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:24.216025 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:24.976865 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:24.976830 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:25.680372 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:25.680333 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:26.289229 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:26.289196 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:26.942108 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:26.942069 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:27.588304 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:27.588264 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:28.238171 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:28.238129 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:28.901972 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:28.901931 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:29.591848 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:29.591814 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:30.212883 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:30.212836 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:30.684755 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:30.684714 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:30.890863 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:30.890829 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:31.000893 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:25:31.000804 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:25:31.198262 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:31.198218 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:25:31.873806 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:31.873766 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:32.483979 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:32.483945 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:33.173023 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:33.172985 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:33.898758 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:33.898697 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:34.564544 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:34.564493 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:35.186572 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:35.186538 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:35.621262 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:35.621171 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-b93c48/simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg"] Oct 31 01:25:35.717276 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:35.717238 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:35.861006 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:35.860966 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:36.584674 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:36.584641 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:37.199697 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:37.199648 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:37.877159 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:37.877125 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:38.605755 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:38.605716 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:39.290138 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:39.290093 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:39.893052 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:39.893009 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:40.745787 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:40.745741 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:41.106864 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:41.106752 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:41.218315 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:41.218265 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:41.397325 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:41.397240 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:41.844996 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:41.844957 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b93c48-77dc6785cc-t8dxx"] Oct 31 01:25:41.845290 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:41.845253 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b93c48-77dc6785cc-t8dxx" podUID="eafc1139-7ac3-44b9-8198-2bf94485dc75" containerName="maas-api" containerID="cri-o://ea4d23e7c952840eef751ff5ce04995b1c00bbb2d0fe6e2ed618159eeee345d4" gracePeriod=30 Oct 31 01:25:41.914982 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:41.914937 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:42.063442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:42.060994 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:42.178231 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:42.178193 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:42.322133 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:42.322104 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b93c48-77dc6785cc-t8dxx" Oct 31 01:25:42.463284 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:42.463246 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eafc1139-7ac3-44b9-8198-2bf94485dc75-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eafc1139-7ac3-44b9-8198-2bf94485dc75-openshift-service-ca\") pod \"eafc1139-7ac3-44b9-8198-2bf94485dc75\" (UID: \"eafc1139-7ac3-44b9-8198-2bf94485dc75\") " Oct 31 01:25:42.463543 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:42.463316 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eafc1139-7ac3-44b9-8198-2bf94485dc75-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eafc1139-7ac3-44b9-8198-2bf94485dc75-maas-api-tls\") pod \"eafc1139-7ac3-44b9-8198-2bf94485dc75\" (UID: \"eafc1139-7ac3-44b9-8198-2bf94485dc75\") " Oct 31 01:25:42.463543 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:42.463386 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eafc1139-7ac3-44b9-8198-2bf94485dc75-kube-api-access-8bw9r\" (UniqueName: \"kubernetes.io/projected/eafc1139-7ac3-44b9-8198-2bf94485dc75-kube-api-access-8bw9r\") pod \"eafc1139-7ac3-44b9-8198-2bf94485dc75\" (UID: \"eafc1139-7ac3-44b9-8198-2bf94485dc75\") " Oct 31 01:25:42.463543 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:42.463448 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eafc1139-7ac3-44b9-8198-2bf94485dc75-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eafc1139-7ac3-44b9-8198-2bf94485dc75-metrics-certs\") pod \"eafc1139-7ac3-44b9-8198-2bf94485dc75\" (UID: \"eafc1139-7ac3-44b9-8198-2bf94485dc75\") " Oct 31 01:25:42.463723 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:42.463657 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eafc1139-7ac3-44b9-8198-2bf94485dc75-openshift-service-ca" pod "eafc1139-7ac3-44b9-8198-2bf94485dc75" (UID: "eafc1139-7ac3-44b9-8198-2bf94485dc75"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:25:42.463898 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:42.463857 4440 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eafc1139-7ac3-44b9-8198-2bf94485dc75-openshift-service-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:25:42.465771 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:42.465734 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eafc1139-7ac3-44b9-8198-2bf94485dc75-kube-api-access-8bw9r" pod "eafc1139-7ac3-44b9-8198-2bf94485dc75" (UID: "eafc1139-7ac3-44b9-8198-2bf94485dc75"). InnerVolumeSpecName "kube-api-access-8bw9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:25:42.465883 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:42.465852 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eafc1139-7ac3-44b9-8198-2bf94485dc75-metrics-certs" pod "eafc1139-7ac3-44b9-8198-2bf94485dc75" (UID: "eafc1139-7ac3-44b9-8198-2bf94485dc75"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:25:42.465971 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:42.465948 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eafc1139-7ac3-44b9-8198-2bf94485dc75-maas-api-tls" pod "eafc1139-7ac3-44b9-8198-2bf94485dc75" (UID: "eafc1139-7ac3-44b9-8198-2bf94485dc75"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:25:42.565401 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:42.565357 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eafc1139-7ac3-44b9-8198-2bf94485dc75-maas-api-tls\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:25:42.565401 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:42.565395 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bw9r\" (UniqueName: \"kubernetes.io/projected/eafc1139-7ac3-44b9-8198-2bf94485dc75-kube-api-access-8bw9r\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:25:42.565401 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:42.565405 4440 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eafc1139-7ac3-44b9-8198-2bf94485dc75-metrics-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:25:42.723198 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:42.723074 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:42.726276 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:42.726243 4440 generic.go:356] "Generic (PLEG): container finished" podID="eafc1139-7ac3-44b9-8198-2bf94485dc75" containerID="ea4d23e7c952840eef751ff5ce04995b1c00bbb2d0fe6e2ed618159eeee345d4" exitCode=0 Oct 31 01:25:42.726473 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:42.726281 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b93c48-77dc6785cc-t8dxx" Oct 31 01:25:42.726473 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:42.726318 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b93c48-77dc6785cc-t8dxx" event={"ID":"eafc1139-7ac3-44b9-8198-2bf94485dc75","Type":"ContainerDied","Data":"ea4d23e7c952840eef751ff5ce04995b1c00bbb2d0fe6e2ed618159eeee345d4"} Oct 31 01:25:42.726473 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:42.726390 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b93c48-77dc6785cc-t8dxx" event={"ID":"eafc1139-7ac3-44b9-8198-2bf94485dc75","Type":"ContainerDied","Data":"746582e19d8a159509c30ea5b48e66b12c3a47fece1538e904d2643b59e636b4"} Oct 31 01:25:42.726473 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:42.726435 4440 scope.go:122] "RemoveContainer" containerID="ea4d23e7c952840eef751ff5ce04995b1c00bbb2d0fe6e2ed618159eeee345d4" Oct 31 01:25:42.737453 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:42.737412 4440 scope.go:122] "RemoveContainer" containerID="ea4d23e7c952840eef751ff5ce04995b1c00bbb2d0fe6e2ed618159eeee345d4" Oct 31 01:25:42.737777 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:25:42.737756 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea4d23e7c952840eef751ff5ce04995b1c00bbb2d0fe6e2ed618159eeee345d4\": container with ID starting with ea4d23e7c952840eef751ff5ce04995b1c00bbb2d0fe6e2ed618159eeee345d4 not found: ID does not exist" containerID="ea4d23e7c952840eef751ff5ce04995b1c00bbb2d0fe6e2ed618159eeee345d4" Oct 31 01:25:42.737868 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:42.737787 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea4d23e7c952840eef751ff5ce04995b1c00bbb2d0fe6e2ed618159eeee345d4"} err="failed to get container status \"ea4d23e7c952840eef751ff5ce04995b1c00bbb2d0fe6e2ed618159eeee345d4\": rpc error: code = NotFound desc = could not find container \"ea4d23e7c952840eef751ff5ce04995b1c00bbb2d0fe6e2ed618159eeee345d4\": container with ID starting with ea4d23e7c952840eef751ff5ce04995b1c00bbb2d0fe6e2ed618159eeee345d4 not found: ID does not exist" Oct 31 01:25:42.752684 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:42.752637 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b93c48-77dc6785cc-t8dxx"] Oct 31 01:25:42.758873 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:42.758843 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b93c48-77dc6785cc-t8dxx"] Oct 31 01:25:43.773254 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:43.773218 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eafc1139-7ac3-44b9-8198-2bf94485dc75" path="/var/lib/kubelet/pods/eafc1139-7ac3-44b9-8198-2bf94485dc75/volumes" Oct 31 01:25:43.797157 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:43.797120 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:25:47.265608 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:47.265569 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:48.499434 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:48.499380 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:48.613218 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:48.613181 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:48.784972 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:48.784872 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz"] Oct 31 01:25:48.785189 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:48.785163 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" podUID="2ccb6a7f-bc35-4a1b-b40d-8255cd731f19" containerName="istio-proxy" containerID="cri-o://3d505a1c1bfa3bb5f9d4e3f595f59c63b0819eac5124d5a55ede0d9b67e8fb00" gracePeriod=30 Oct 31 01:25:48.932647 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:48.932608 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:50.642122 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:50.642055 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-b93c48/simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg" podUID="ecca2fe3-c9c5-4d53-a4a1-96d9dce86635" containerName="main" containerID="cri-o://3fc6ae618752585150302663816b1babb114715d764d36e555b2b38efd0344f2" gracePeriod=45 Oct 31 01:25:50.794672 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:50.794639 4440 generic.go:356] "Generic (PLEG): container finished" podID="ecca2fe3-c9c5-4d53-a4a1-96d9dce86635" containerID="3fc6ae618752585150302663816b1babb114715d764d36e555b2b38efd0344f2" exitCode=0 Oct 31 01:25:50.794860 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:50.794679 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b93c48/simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg" event={"ID":"ecca2fe3-c9c5-4d53-a4a1-96d9dce86635","Type":"ContainerDied","Data":"3fc6ae618752585150302663816b1babb114715d764d36e555b2b38efd0344f2"} Oct 31 01:25:50.903067 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:50.903000 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b93c48/simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg" Oct 31 01:25:50.953481 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:50.953443 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635-model-cache\") pod \"ecca2fe3-c9c5-4d53-a4a1-96d9dce86635\" (UID: \"ecca2fe3-c9c5-4d53-a4a1-96d9dce86635\") " Oct 31 01:25:50.953683 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:50.953505 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635-home\" (UniqueName: \"kubernetes.io/empty-dir/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635-home\") pod \"ecca2fe3-c9c5-4d53-a4a1-96d9dce86635\" (UID: \"ecca2fe3-c9c5-4d53-a4a1-96d9dce86635\") " Oct 31 01:25:50.953683 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:50.953588 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635-dshm\") pod \"ecca2fe3-c9c5-4d53-a4a1-96d9dce86635\" (UID: \"ecca2fe3-c9c5-4d53-a4a1-96d9dce86635\") " Oct 31 01:25:50.953683 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:50.953666 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635-tmp-dir\") pod \"ecca2fe3-c9c5-4d53-a4a1-96d9dce86635\" (UID: \"ecca2fe3-c9c5-4d53-a4a1-96d9dce86635\") " Oct 31 01:25:50.953883 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:50.953706 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635-tls-certs\" (UniqueName: \"kubernetes.io/secret/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635-tls-certs\") pod \"ecca2fe3-c9c5-4d53-a4a1-96d9dce86635\" (UID: \"ecca2fe3-c9c5-4d53-a4a1-96d9dce86635\") " Oct 31 01:25:50.953883 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:50.953738 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635-model-cache" pod "ecca2fe3-c9c5-4d53-a4a1-96d9dce86635" (UID: "ecca2fe3-c9c5-4d53-a4a1-96d9dce86635"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:25:50.953883 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:50.953869 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635-home" pod "ecca2fe3-c9c5-4d53-a4a1-96d9dce86635" (UID: "ecca2fe3-c9c5-4d53-a4a1-96d9dce86635"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:25:50.954039 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:50.953876 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635-kube-api-access-nq966\" (UniqueName: \"kubernetes.io/projected/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635-kube-api-access-nq966\") pod \"ecca2fe3-c9c5-4d53-a4a1-96d9dce86635\" (UID: \"ecca2fe3-c9c5-4d53-a4a1-96d9dce86635\") " Oct 31 01:25:50.954039 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:50.953923 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635-tmp-dir" pod "ecca2fe3-c9c5-4d53-a4a1-96d9dce86635" (UID: "ecca2fe3-c9c5-4d53-a4a1-96d9dce86635"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:25:50.954253 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:50.954233 4440 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635-tmp-dir\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:25:50.954321 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:50.954260 4440 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635-model-cache\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:25:50.954321 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:50.954274 4440 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635-home\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:25:50.956214 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:50.956181 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635-tls-certs" pod "ecca2fe3-c9c5-4d53-a4a1-96d9dce86635" (UID: "ecca2fe3-c9c5-4d53-a4a1-96d9dce86635"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:25:50.956343 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:50.956256 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635-dshm" pod "ecca2fe3-c9c5-4d53-a4a1-96d9dce86635" (UID: "ecca2fe3-c9c5-4d53-a4a1-96d9dce86635"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:25:50.956343 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:50.956293 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635-kube-api-access-nq966" pod "ecca2fe3-c9c5-4d53-a4a1-96d9dce86635" (UID: "ecca2fe3-c9c5-4d53-a4a1-96d9dce86635"). InnerVolumeSpecName "kube-api-access-nq966". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:25:51.055309 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:51.055259 4440 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635-dshm\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:25:51.055309 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:51.055299 4440 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635-tls-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:25:51.055570 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:51.055319 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq966\" (UniqueName: \"kubernetes.io/projected/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635-kube-api-access-nq966\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:25:51.801933 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:51.801901 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b93c48/simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg" Oct 31 01:25:51.802443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:51.801946 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b93c48/simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg" event={"ID":"ecca2fe3-c9c5-4d53-a4a1-96d9dce86635","Type":"ContainerDied","Data":"d95545164d962c5e886264794c8c9c2b49d08dda3763109a3273790b7bd1f75f"} Oct 31 01:25:51.802443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:51.802005 4440 scope.go:122] "RemoveContainer" containerID="3fc6ae618752585150302663816b1babb114715d764d36e555b2b38efd0344f2" Oct 31 01:25:51.818428 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:51.818391 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-b93c48/simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg"] Oct 31 01:25:51.822136 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:51.822108 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-b93c48/simulator-single-main-b93c48-kserve-5d95fd6977-hs2sg"] Oct 31 01:25:51.932033 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:51.931989 4440 patch_prober.go:29] interesting pod/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:25:51.932198 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:51.932061 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" podUID="2ccb6a7f-bc35-4a1b-b40d-8255cd731f19" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:25:53.770831 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:53.770786 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecca2fe3-c9c5-4d53-a4a1-96d9dce86635" path="/var/lib/kubelet/pods/ecca2fe3-c9c5-4d53-a4a1-96d9dce86635/volumes" Oct 31 01:25:53.856181 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:53.856146 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg"] Oct 31 01:25:53.856619 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:53.856605 4440 cpu_manager.go:414] "containerMap: removing container" podUID="ecca2fe3-c9c5-4d53-a4a1-96d9dce86635" containerName="main" Oct 31 01:25:53.856619 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:53.856619 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecca2fe3-c9c5-4d53-a4a1-96d9dce86635" containerName="main" Oct 31 01:25:53.856761 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:53.856635 4440 cpu_manager.go:414] "containerMap: removing container" podUID="eafc1139-7ac3-44b9-8198-2bf94485dc75" containerName="maas-api" Oct 31 01:25:53.856761 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:53.856640 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eafc1139-7ac3-44b9-8198-2bf94485dc75" containerName="maas-api" Oct 31 01:25:53.856761 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:53.856697 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="eafc1139-7ac3-44b9-8198-2bf94485dc75" containerName="maas-api" Oct 31 01:25:53.856761 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:53.856712 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecca2fe3-c9c5-4d53-a4a1-96d9dce86635" containerName="main" Oct 31 01:25:53.860708 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:53.860684 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg" Oct 31 01:25:53.867470 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:53.867443 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg"] Oct 31 01:25:53.870055 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:53.870029 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-1dd35abe-kserve-self-signed-certs\"" Oct 31 01:25:53.885246 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:53.885213 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8762b195-312d-431c-bca2-c9a6dc064f6e-home\") pod \"e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg\" (UID: \"8762b195-312d-431c-bca2-c9a6dc064f6e\") " pod="llm/e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg" Oct 31 01:25:53.885463 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:53.885285 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnsjr\" (UniqueName: \"kubernetes.io/projected/8762b195-312d-431c-bca2-c9a6dc064f6e-kube-api-access-dnsjr\") pod \"e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg\" (UID: \"8762b195-312d-431c-bca2-c9a6dc064f6e\") " pod="llm/e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg" Oct 31 01:25:53.885463 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:53.885397 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8762b195-312d-431c-bca2-c9a6dc064f6e-model-cache\") pod \"e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg\" (UID: \"8762b195-312d-431c-bca2-c9a6dc064f6e\") " pod="llm/e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg" Oct 31 01:25:53.885610 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:53.885551 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8762b195-312d-431c-bca2-c9a6dc064f6e-dshm\") pod \"e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg\" (UID: \"8762b195-312d-431c-bca2-c9a6dc064f6e\") " pod="llm/e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg" Oct 31 01:25:53.885610 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:53.885587 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8762b195-312d-431c-bca2-c9a6dc064f6e-tmp-dir\") pod \"e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg\" (UID: \"8762b195-312d-431c-bca2-c9a6dc064f6e\") " pod="llm/e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg" Oct 31 01:25:53.886083 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:53.885713 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8762b195-312d-431c-bca2-c9a6dc064f6e-tls-certs\") pod \"e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg\" (UID: \"8762b195-312d-431c-bca2-c9a6dc064f6e\") " pod="llm/e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg" Oct 31 01:25:53.928138 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:53.928076 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:53.987321 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:53.987048 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8762b195-312d-431c-bca2-c9a6dc064f6e-dshm\") pod \"e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg\" (UID: \"8762b195-312d-431c-bca2-c9a6dc064f6e\") " pod="llm/e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg" Oct 31 01:25:53.987321 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:53.987113 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8762b195-312d-431c-bca2-c9a6dc064f6e-tmp-dir\") pod \"e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg\" (UID: \"8762b195-312d-431c-bca2-c9a6dc064f6e\") " pod="llm/e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg" Oct 31 01:25:53.987321 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:53.987170 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8762b195-312d-431c-bca2-c9a6dc064f6e-tls-certs\") pod \"e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg\" (UID: \"8762b195-312d-431c-bca2-c9a6dc064f6e\") " pod="llm/e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg" Oct 31 01:25:53.987321 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:53.987252 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8762b195-312d-431c-bca2-c9a6dc064f6e-home\") pod \"e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg\" (UID: \"8762b195-312d-431c-bca2-c9a6dc064f6e\") " pod="llm/e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg" Oct 31 01:25:53.987321 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:53.987316 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnsjr\" (UniqueName: \"kubernetes.io/projected/8762b195-312d-431c-bca2-c9a6dc064f6e-kube-api-access-dnsjr\") pod \"e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg\" (UID: \"8762b195-312d-431c-bca2-c9a6dc064f6e\") " pod="llm/e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg" Oct 31 01:25:53.987846 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:53.987354 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8762b195-312d-431c-bca2-c9a6dc064f6e-model-cache\") pod \"e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg\" (UID: \"8762b195-312d-431c-bca2-c9a6dc064f6e\") " pod="llm/e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg" Oct 31 01:25:53.987920 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:53.987854 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8762b195-312d-431c-bca2-c9a6dc064f6e-model-cache\") pod \"e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg\" (UID: \"8762b195-312d-431c-bca2-c9a6dc064f6e\") " pod="llm/e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg" Oct 31 01:25:53.990332 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:53.990302 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8762b195-312d-431c-bca2-c9a6dc064f6e-tmp-dir\") pod \"e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg\" (UID: \"8762b195-312d-431c-bca2-c9a6dc064f6e\") " pod="llm/e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg" Oct 31 01:25:54.041085 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.040988 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8762b195-312d-431c-bca2-c9a6dc064f6e-dshm\") pod \"e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg\" (UID: \"8762b195-312d-431c-bca2-c9a6dc064f6e\") " pod="llm/e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg" Oct 31 01:25:54.041500 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.041459 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8762b195-312d-431c-bca2-c9a6dc064f6e-home\") pod \"e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg\" (UID: \"8762b195-312d-431c-bca2-c9a6dc064f6e\") " pod="llm/e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg" Oct 31 01:25:54.045038 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.045015 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8762b195-312d-431c-bca2-c9a6dc064f6e-tls-certs\") pod \"e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg\" (UID: \"8762b195-312d-431c-bca2-c9a6dc064f6e\") " pod="llm/e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg" Oct 31 01:25:54.054275 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.054245 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnsjr\" (UniqueName: \"kubernetes.io/projected/8762b195-312d-431c-bca2-c9a6dc064f6e-kube-api-access-dnsjr\") pod \"e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg\" (UID: \"8762b195-312d-431c-bca2-c9a6dc064f6e\") " pod="llm/e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg" Oct 31 01:25:54.080150 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.080120 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" Oct 31 01:25:54.175783 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.175742 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg" Oct 31 01:25:54.188900 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.188868 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-kube-api-access-2xqw2\" (UniqueName: \"kubernetes.io/projected/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-kube-api-access-2xqw2\") pod \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\" (UID: \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\") " Oct 31 01:25:54.188973 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.188936 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-workload-socket\") pod \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\" (UID: \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\") " Oct 31 01:25:54.189047 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.189032 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-workload-certs\") pod \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\" (UID: \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\") " Oct 31 01:25:54.189091 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.189067 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-istio-token\" (UniqueName: \"kubernetes.io/projected/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-istio-token\") pod \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\" (UID: \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\") " Oct 31 01:25:54.189156 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.189098 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-credential-socket\") pod \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\" (UID: \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\") " Oct 31 01:25:54.189156 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.189128 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-istiod-ca-cert\") pod \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\" (UID: \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\") " Oct 31 01:25:54.189265 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.189169 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-istio-data\") pod \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\" (UID: \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\") " Oct 31 01:25:54.189265 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.189236 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-istio-podinfo\") pod \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\" (UID: \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\") " Oct 31 01:25:54.189265 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.189231 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-workload-socket" pod "2ccb6a7f-bc35-4a1b-b40d-8255cd731f19" (UID: "2ccb6a7f-bc35-4a1b-b40d-8255cd731f19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:25:54.189508 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.189294 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-istio-envoy\") pod \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\" (UID: \"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19\") " Oct 31 01:25:54.189508 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.189318 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-workload-certs" pod "2ccb6a7f-bc35-4a1b-b40d-8255cd731f19" (UID: "2ccb6a7f-bc35-4a1b-b40d-8255cd731f19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:25:54.189630 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.189560 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-istio-data" pod "2ccb6a7f-bc35-4a1b-b40d-8255cd731f19" (UID: "2ccb6a7f-bc35-4a1b-b40d-8255cd731f19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:25:54.189630 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.189588 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-istiod-ca-cert" pod "2ccb6a7f-bc35-4a1b-b40d-8255cd731f19" (UID: "2ccb6a7f-bc35-4a1b-b40d-8255cd731f19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:25:54.189788 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.189765 4440 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-workload-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:25:54.189863 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.189798 4440 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-workload-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:25:54.189863 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.189813 4440 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-istiod-ca-cert\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:25:54.189863 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.189827 4440 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-istio-data\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:25:54.189863 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.189767 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-credential-socket" pod "2ccb6a7f-bc35-4a1b-b40d-8255cd731f19" (UID: "2ccb6a7f-bc35-4a1b-b40d-8255cd731f19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:25:54.191482 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.191456 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-kube-api-access-2xqw2" pod "2ccb6a7f-bc35-4a1b-b40d-8255cd731f19" (UID: "2ccb6a7f-bc35-4a1b-b40d-8255cd731f19"). InnerVolumeSpecName "kube-api-access-2xqw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:25:54.191696 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.191675 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-istio-token" pod "2ccb6a7f-bc35-4a1b-b40d-8255cd731f19" (UID: "2ccb6a7f-bc35-4a1b-b40d-8255cd731f19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:25:54.192244 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.192228 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-istio-envoy" pod "2ccb6a7f-bc35-4a1b-b40d-8255cd731f19" (UID: "2ccb6a7f-bc35-4a1b-b40d-8255cd731f19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:25:54.192361 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.192338 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-istio-podinfo" pod "2ccb6a7f-bc35-4a1b-b40d-8255cd731f19" (UID: "2ccb6a7f-bc35-4a1b-b40d-8255cd731f19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:25:54.291234 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.291148 4440 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-istio-token\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:25:54.291234 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.291181 4440 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-credential-socket\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:25:54.291234 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.291190 4440 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-istio-podinfo\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:25:54.291234 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.291199 4440 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-istio-envoy\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:25:54.291234 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.291208 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xqw2\" (UniqueName: \"kubernetes.io/projected/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19-kube-api-access-2xqw2\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:25:54.456339 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.456307 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg"] Oct 31 01:25:54.456616 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:25:54.456591 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8762b195_312d_431c_bca2_c9a6dc064f6e.slice/crio-b301064ee3ba4f676f49b6447155cf3ab955e1765d2c8e1dd7e6bc83bde42ca4 WatchSource:0}: Error finding container b301064ee3ba4f676f49b6447155cf3ab955e1765d2c8e1dd7e6bc83bde42ca4: Status 404 returned error can't find the container with id b301064ee3ba4f676f49b6447155cf3ab955e1765d2c8e1dd7e6bc83bde42ca4 Oct 31 01:25:54.824074 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.824027 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg" event={"ID":"8762b195-312d-431c-bca2-c9a6dc064f6e","Type":"ContainerStarted","Data":"6b80922b2e69b482dc2ec504f3620dfafbaaac836b369b4f2bc377e04bd54a23"} Oct 31 01:25:54.824543 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.824090 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg" event={"ID":"8762b195-312d-431c-bca2-c9a6dc064f6e","Type":"ContainerStarted","Data":"b301064ee3ba4f676f49b6447155cf3ab955e1765d2c8e1dd7e6bc83bde42ca4"} Oct 31 01:25:54.826051 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.826022 4440 generic.go:356] "Generic (PLEG): container finished" podID="2ccb6a7f-bc35-4a1b-b40d-8255cd731f19" containerID="3d505a1c1bfa3bb5f9d4e3f595f59c63b0819eac5124d5a55ede0d9b67e8fb00" exitCode=0 Oct 31 01:25:54.826178 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.826062 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" Oct 31 01:25:54.826178 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.826065 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" event={"ID":"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19","Type":"ContainerDied","Data":"3d505a1c1bfa3bb5f9d4e3f595f59c63b0819eac5124d5a55ede0d9b67e8fb00"} Oct 31 01:25:54.826263 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.826193 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz" event={"ID":"2ccb6a7f-bc35-4a1b-b40d-8255cd731f19","Type":"ContainerDied","Data":"b4c7dce401d399c3ae2794c77a2566c5095767cd585ac735531825e9b291f51a"} Oct 31 01:25:54.826263 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.826213 4440 scope.go:122] "RemoveContainer" containerID="3d505a1c1bfa3bb5f9d4e3f595f59c63b0819eac5124d5a55ede0d9b67e8fb00" Oct 31 01:25:54.836985 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.836965 4440 scope.go:122] "RemoveContainer" containerID="735d187006bb576f1734d44a6d73fb00e20d658165e52fbdb76d047637aecc87" Oct 31 01:25:54.845876 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.845819 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg" podStartSLOduration=1.845799384 podStartE2EDuration="1.845799384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:25:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:25:54.840229566 +0000 UTC m=+1799.198951884" watchObservedRunningTime="2026-10-31 01:25:54.845799384 +0000 UTC m=+1799.204521702" Oct 31 01:25:54.848047 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.848022 4440 scope.go:122] "RemoveContainer" containerID="3d505a1c1bfa3bb5f9d4e3f595f59c63b0819eac5124d5a55ede0d9b67e8fb00" Oct 31 01:25:54.848399 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:25:54.848378 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d505a1c1bfa3bb5f9d4e3f595f59c63b0819eac5124d5a55ede0d9b67e8fb00\": container with ID starting with 3d505a1c1bfa3bb5f9d4e3f595f59c63b0819eac5124d5a55ede0d9b67e8fb00 not found: ID does not exist" containerID="3d505a1c1bfa3bb5f9d4e3f595f59c63b0819eac5124d5a55ede0d9b67e8fb00" Oct 31 01:25:54.848462 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.848410 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d505a1c1bfa3bb5f9d4e3f595f59c63b0819eac5124d5a55ede0d9b67e8fb00"} err="failed to get container status \"3d505a1c1bfa3bb5f9d4e3f595f59c63b0819eac5124d5a55ede0d9b67e8fb00\": rpc error: code = NotFound desc = could not find container \"3d505a1c1bfa3bb5f9d4e3f595f59c63b0819eac5124d5a55ede0d9b67e8fb00\": container with ID starting with 3d505a1c1bfa3bb5f9d4e3f595f59c63b0819eac5124d5a55ede0d9b67e8fb00 not found: ID does not exist" Oct 31 01:25:54.848462 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.848447 4440 scope.go:122] "RemoveContainer" containerID="735d187006bb576f1734d44a6d73fb00e20d658165e52fbdb76d047637aecc87" Oct 31 01:25:54.848758 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:25:54.848736 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"735d187006bb576f1734d44a6d73fb00e20d658165e52fbdb76d047637aecc87\": container with ID starting with 735d187006bb576f1734d44a6d73fb00e20d658165e52fbdb76d047637aecc87 not found: ID does not exist" containerID="735d187006bb576f1734d44a6d73fb00e20d658165e52fbdb76d047637aecc87" Oct 31 01:25:54.848881 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.848762 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"735d187006bb576f1734d44a6d73fb00e20d658165e52fbdb76d047637aecc87"} err="failed to get container status \"735d187006bb576f1734d44a6d73fb00e20d658165e52fbdb76d047637aecc87\": rpc error: code = NotFound desc = could not find container \"735d187006bb576f1734d44a6d73fb00e20d658165e52fbdb76d047637aecc87\": container with ID starting with 735d187006bb576f1734d44a6d73fb00e20d658165e52fbdb76d047637aecc87 not found: ID does not exist" Oct 31 01:25:54.856373 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.856342 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz"] Oct 31 01:25:54.860363 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:54.860332 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b93c48-openshift-default-678fcbf889kjtqz"] Oct 31 01:25:55.175982 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:55.175952 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg" Oct 31 01:25:55.493864 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:55.493776 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:25:55.774280 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:55.774201 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ccb6a7f-bc35-4a1b-b40d-8255cd731f19" path="/var/lib/kubelet/pods/2ccb6a7f-bc35-4a1b-b40d-8255cd731f19/volumes" Oct 31 01:25:56.950000 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:56.949959 4440 scope.go:122] "RemoveContainer" containerID="4c1fc1ead1f7bb25b9eef566e813be40847cb1c1087e2c36c27d637eaf3a6eb5" Oct 31 01:25:56.964264 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:56.964234 4440 scope.go:122] "RemoveContainer" containerID="a53d3ff1d982ef5c049bb2c8bd2418cf486c94666733c6f4445746c6396297e2" Oct 31 01:25:56.978918 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:25:56.978883 4440 scope.go:122] "RemoveContainer" containerID="f4431a0fe49c22f8a9a008a679002d680737f5160c684e9a4783522afdf557dd" Oct 31 01:26:01.018643 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:26:01.018593 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:26:04.176296 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:04.176260 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg" Oct 31 01:26:04.188524 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:04.188495 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg" Oct 31 01:26:05.074773 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:05.074736 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg" Oct 31 01:26:09.105586 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:09.105544 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl"] Oct 31 01:26:09.106080 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:09.105946 4440 cpu_manager.go:414] "containerMap: removing container" podUID="2ccb6a7f-bc35-4a1b-b40d-8255cd731f19" containerName="istio-proxy" Oct 31 01:26:09.106080 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:09.105957 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ccb6a7f-bc35-4a1b-b40d-8255cd731f19" containerName="istio-proxy" Oct 31 01:26:09.106080 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:09.106036 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ccb6a7f-bc35-4a1b-b40d-8255cd731f19" containerName="istio-proxy" Oct 31 01:26:09.106080 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:09.106047 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ccb6a7f-bc35-4a1b-b40d-8255cd731f19" containerName="istio-proxy" Oct 31 01:26:09.110916 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:09.110892 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl" Oct 31 01:26:09.120979 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:09.120958 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-1dd35abe-kserve-self-signed-certs\"" Oct 31 01:26:09.122024 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:09.121988 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl"] Oct 31 01:26:09.220838 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:09.220806 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/465bc027-ddc4-4f02-9d78-700e8578d9aa-tmp-dir\") pod \"e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl\" (UID: \"465bc027-ddc4-4f02-9d78-700e8578d9aa\") " pod="llm/e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl" Oct 31 01:26:09.221007 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:09.220849 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl2ng\" (UniqueName: \"kubernetes.io/projected/465bc027-ddc4-4f02-9d78-700e8578d9aa-kube-api-access-zl2ng\") pod \"e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl\" (UID: \"465bc027-ddc4-4f02-9d78-700e8578d9aa\") " pod="llm/e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl" Oct 31 01:26:09.221007 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:09.220972 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/465bc027-ddc4-4f02-9d78-700e8578d9aa-dshm\") pod \"e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl\" (UID: \"465bc027-ddc4-4f02-9d78-700e8578d9aa\") " pod="llm/e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl" Oct 31 01:26:09.221121 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:09.221011 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/465bc027-ddc4-4f02-9d78-700e8578d9aa-home\") pod \"e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl\" (UID: \"465bc027-ddc4-4f02-9d78-700e8578d9aa\") " pod="llm/e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl" Oct 31 01:26:09.221121 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:09.221033 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/465bc027-ddc4-4f02-9d78-700e8578d9aa-model-cache\") pod \"e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl\" (UID: \"465bc027-ddc4-4f02-9d78-700e8578d9aa\") " pod="llm/e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl" Oct 31 01:26:09.221228 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:09.221165 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/465bc027-ddc4-4f02-9d78-700e8578d9aa-tls-certs\") pod \"e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl\" (UID: \"465bc027-ddc4-4f02-9d78-700e8578d9aa\") " pod="llm/e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl" Oct 31 01:26:09.271722 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:09.271681 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:26:09.322280 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:09.322250 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/465bc027-ddc4-4f02-9d78-700e8578d9aa-home\") pod \"e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl\" (UID: \"465bc027-ddc4-4f02-9d78-700e8578d9aa\") " pod="llm/e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl" Oct 31 01:26:09.322280 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:09.322285 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/465bc027-ddc4-4f02-9d78-700e8578d9aa-model-cache\") pod \"e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl\" (UID: \"465bc027-ddc4-4f02-9d78-700e8578d9aa\") " pod="llm/e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl" Oct 31 01:26:09.322516 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:09.322315 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/465bc027-ddc4-4f02-9d78-700e8578d9aa-tls-certs\") pod \"e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl\" (UID: \"465bc027-ddc4-4f02-9d78-700e8578d9aa\") " pod="llm/e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl" Oct 31 01:26:09.322516 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:09.322375 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/465bc027-ddc4-4f02-9d78-700e8578d9aa-tmp-dir\") pod \"e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl\" (UID: \"465bc027-ddc4-4f02-9d78-700e8578d9aa\") " pod="llm/e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl" Oct 31 01:26:09.322516 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:09.322400 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl2ng\" (UniqueName: \"kubernetes.io/projected/465bc027-ddc4-4f02-9d78-700e8578d9aa-kube-api-access-zl2ng\") pod \"e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl\" (UID: \"465bc027-ddc4-4f02-9d78-700e8578d9aa\") " pod="llm/e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl" Oct 31 01:26:09.322516 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:09.322490 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/465bc027-ddc4-4f02-9d78-700e8578d9aa-dshm\") pod \"e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl\" (UID: \"465bc027-ddc4-4f02-9d78-700e8578d9aa\") " pod="llm/e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl" Oct 31 01:26:09.322759 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:09.322725 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/465bc027-ddc4-4f02-9d78-700e8578d9aa-model-cache\") pod \"e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl\" (UID: \"465bc027-ddc4-4f02-9d78-700e8578d9aa\") " pod="llm/e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl" Oct 31 01:26:09.322829 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:09.322756 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/465bc027-ddc4-4f02-9d78-700e8578d9aa-home\") pod \"e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl\" (UID: \"465bc027-ddc4-4f02-9d78-700e8578d9aa\") " pod="llm/e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl" Oct 31 01:26:09.322829 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:09.322768 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/465bc027-ddc4-4f02-9d78-700e8578d9aa-tmp-dir\") pod \"e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl\" (UID: \"465bc027-ddc4-4f02-9d78-700e8578d9aa\") " pod="llm/e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl" Oct 31 01:26:09.324669 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:09.324646 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/465bc027-ddc4-4f02-9d78-700e8578d9aa-dshm\") pod \"e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl\" (UID: \"465bc027-ddc4-4f02-9d78-700e8578d9aa\") " pod="llm/e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl" Oct 31 01:26:09.324846 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:09.324827 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/465bc027-ddc4-4f02-9d78-700e8578d9aa-tls-certs\") pod \"e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl\" (UID: \"465bc027-ddc4-4f02-9d78-700e8578d9aa\") " pod="llm/e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl" Oct 31 01:26:09.330014 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:09.329991 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl2ng\" (UniqueName: \"kubernetes.io/projected/465bc027-ddc4-4f02-9d78-700e8578d9aa-kube-api-access-zl2ng\") pod \"e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl\" (UID: \"465bc027-ddc4-4f02-9d78-700e8578d9aa\") " pod="llm/e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl" Oct 31 01:26:09.425255 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:09.425224 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl" Oct 31 01:26:09.715588 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:09.710465 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl"] Oct 31 01:26:10.092462 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:10.092336 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl" event={"ID":"465bc027-ddc4-4f02-9d78-700e8578d9aa","Type":"ContainerStarted","Data":"77f6b06e36ce74e6d7265f4a645cb19295da9f72448345835845945a4dfff2f6"} Oct 31 01:26:10.092462 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:10.092388 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl" event={"ID":"465bc027-ddc4-4f02-9d78-700e8578d9aa","Type":"ContainerStarted","Data":"1f424b0eb4b46b43d11ce71ace19c517d705cb671f5bca130285d10535e2493d"} Oct 31 01:26:10.110473 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:10.110397 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl" podStartSLOduration=1.110378275 podStartE2EDuration="1.110378275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:26:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:26:10.108144235 +0000 UTC m=+1814.466866552" watchObservedRunningTime="2026-10-31 01:26:10.110378275 +0000 UTC m=+1814.469100591" Oct 31 01:26:10.425407 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:10.425373 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl" Oct 31 01:26:10.793488 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:10.793383 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:26:12.194930 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:12.194889 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:26:12.739443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:12.722051 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:26:18.243593 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:18.243560 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:26:19.425776 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:19.425731 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl" Oct 31 01:26:19.438455 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:19.438408 4440 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl" Oct 31 01:26:20.228334 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:20.228295 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl" Oct 31 01:26:27.286023 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:27.285980 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:26:31.206442 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:26:31.205660 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:26:37.606338 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:37.606257 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg"] Oct 31 01:26:37.710287 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:37.710241 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:26:39.310843 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:39.310805 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:26:39.318443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:39.318384 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl"] Oct 31 01:26:39.374828 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:39.374797 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:26:45.751442 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:45.751265 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:26:46.405611 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:46.405561 4440 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-qhwdc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:26:46.405829 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:46.405651 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qhwdc" podUID="103d9e31-f559-4366-98ab-2c80cb627428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:26:46.469894 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:46.469861 4440 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-wk87j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:26:46.470085 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:46.469914 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-wk87j" podUID="57c830db-af34-4fb2-b933-f85f04afb3c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:26:46.472562 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:46.472525 4440 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-8tjdw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:26:46.472670 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:46.472574 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8tjdw" podUID="56fbc0ea-4056-4f36-867b-68601554c6fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:26:47.236822 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:47.236782 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:26:51.898106 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:51.898074 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:26:52.639443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:52.634790 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg" podUID="8762b195-312d-431c-bca2-c9a6dc064f6e" containerName="main" containerID="cri-o://6b80922b2e69b482dc2ec504f3620dfafbaaac836b369b4f2bc377e04bd54a23" gracePeriod=45 Oct 31 01:26:53.066441 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:53.063826 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg" podUID="8762b195-312d-431c-bca2-c9a6dc064f6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.127:8000/ready\": dial tcp 10.133.0.127:8000: connect: connection refused" Oct 31 01:26:54.049656 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:54.028176 4440 generic.go:356] "Generic (PLEG): container finished" podID="8762b195-312d-431c-bca2-c9a6dc064f6e" containerID="6b80922b2e69b482dc2ec504f3620dfafbaaac836b369b4f2bc377e04bd54a23" exitCode=0 Oct 31 01:26:54.049656 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:54.028324 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg" event={"ID":"8762b195-312d-431c-bca2-c9a6dc064f6e","Type":"ContainerDied","Data":"6b80922b2e69b482dc2ec504f3620dfafbaaac836b369b4f2bc377e04bd54a23"} Oct 31 01:26:54.188715 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:54.188671 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg" Oct 31 01:26:54.339884 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:54.339790 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8762b195-312d-431c-bca2-c9a6dc064f6e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8762b195-312d-431c-bca2-c9a6dc064f6e-tmp-dir\") pod \"8762b195-312d-431c-bca2-c9a6dc064f6e\" (UID: \"8762b195-312d-431c-bca2-c9a6dc064f6e\") " Oct 31 01:26:54.340182 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:54.340158 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8762b195-312d-431c-bca2-c9a6dc064f6e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8762b195-312d-431c-bca2-c9a6dc064f6e-dshm\") pod \"8762b195-312d-431c-bca2-c9a6dc064f6e\" (UID: \"8762b195-312d-431c-bca2-c9a6dc064f6e\") " Oct 31 01:26:54.340361 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:54.340344 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8762b195-312d-431c-bca2-c9a6dc064f6e-tls-certs\" (UniqueName: \"kubernetes.io/secret/8762b195-312d-431c-bca2-c9a6dc064f6e-tls-certs\") pod \"8762b195-312d-431c-bca2-c9a6dc064f6e\" (UID: \"8762b195-312d-431c-bca2-c9a6dc064f6e\") " Oct 31 01:26:54.346794 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:54.346763 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8762b195-312d-431c-bca2-c9a6dc064f6e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8762b195-312d-431c-bca2-c9a6dc064f6e-model-cache\") pod \"8762b195-312d-431c-bca2-c9a6dc064f6e\" (UID: \"8762b195-312d-431c-bca2-c9a6dc064f6e\") " Oct 31 01:26:54.347056 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:54.347033 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8762b195-312d-431c-bca2-c9a6dc064f6e-home\" (UniqueName: \"kubernetes.io/empty-dir/8762b195-312d-431c-bca2-c9a6dc064f6e-home\") pod \"8762b195-312d-431c-bca2-c9a6dc064f6e\" (UID: \"8762b195-312d-431c-bca2-c9a6dc064f6e\") " Oct 31 01:26:54.347216 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:54.347197 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8762b195-312d-431c-bca2-c9a6dc064f6e-kube-api-access-dnsjr\" (UniqueName: \"kubernetes.io/projected/8762b195-312d-431c-bca2-c9a6dc064f6e-kube-api-access-dnsjr\") pod \"8762b195-312d-431c-bca2-c9a6dc064f6e\" (UID: \"8762b195-312d-431c-bca2-c9a6dc064f6e\") " Oct 31 01:26:54.375436 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:54.375137 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8762b195-312d-431c-bca2-c9a6dc064f6e-home" pod "8762b195-312d-431c-bca2-c9a6dc064f6e" (UID: "8762b195-312d-431c-bca2-c9a6dc064f6e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:26:54.375715 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:54.375669 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8762b195-312d-431c-bca2-c9a6dc064f6e-tmp-dir" pod "8762b195-312d-431c-bca2-c9a6dc064f6e" (UID: "8762b195-312d-431c-bca2-c9a6dc064f6e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:26:54.375871 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:54.375848 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8762b195-312d-431c-bca2-c9a6dc064f6e-model-cache" pod "8762b195-312d-431c-bca2-c9a6dc064f6e" (UID: "8762b195-312d-431c-bca2-c9a6dc064f6e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:26:54.376320 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:54.376283 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl" podUID="465bc027-ddc4-4f02-9d78-700e8578d9aa" containerName="main" containerID="cri-o://77f6b06e36ce74e6d7265f4a645cb19295da9f72448345835845945a4dfff2f6" gracePeriod=45 Oct 31 01:26:54.380492 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:54.380468 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8762b195-312d-431c-bca2-c9a6dc064f6e-tls-certs" pod "8762b195-312d-431c-bca2-c9a6dc064f6e" (UID: "8762b195-312d-431c-bca2-c9a6dc064f6e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:26:54.383960 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:54.383937 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8762b195-312d-431c-bca2-c9a6dc064f6e-kube-api-access-dnsjr" pod "8762b195-312d-431c-bca2-c9a6dc064f6e" (UID: "8762b195-312d-431c-bca2-c9a6dc064f6e"). InnerVolumeSpecName "kube-api-access-dnsjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:26:54.384759 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:54.384725 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8762b195-312d-431c-bca2-c9a6dc064f6e-dshm" pod "8762b195-312d-431c-bca2-c9a6dc064f6e" (UID: "8762b195-312d-431c-bca2-c9a6dc064f6e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:26:54.451204 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:54.451079 4440 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8762b195-312d-431c-bca2-c9a6dc064f6e-tls-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:26:54.451204 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:54.451119 4440 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8762b195-312d-431c-bca2-c9a6dc064f6e-model-cache\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:26:54.451204 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:54.451135 4440 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8762b195-312d-431c-bca2-c9a6dc064f6e-home\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:26:54.451204 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:54.451152 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnsjr\" (UniqueName: \"kubernetes.io/projected/8762b195-312d-431c-bca2-c9a6dc064f6e-kube-api-access-dnsjr\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:26:54.451204 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:54.451167 4440 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8762b195-312d-431c-bca2-c9a6dc064f6e-tmp-dir\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:26:54.451204 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:54.451180 4440 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8762b195-312d-431c-bca2-c9a6dc064f6e-dshm\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:26:54.687713 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:54.687685 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl" Oct 31 01:26:54.855627 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:54.855545 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/465bc027-ddc4-4f02-9d78-700e8578d9aa-tls-certs\" (UniqueName: \"kubernetes.io/secret/465bc027-ddc4-4f02-9d78-700e8578d9aa-tls-certs\") pod \"465bc027-ddc4-4f02-9d78-700e8578d9aa\" (UID: \"465bc027-ddc4-4f02-9d78-700e8578d9aa\") " Oct 31 01:26:54.855627 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:54.855614 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/465bc027-ddc4-4f02-9d78-700e8578d9aa-home\" (UniqueName: \"kubernetes.io/empty-dir/465bc027-ddc4-4f02-9d78-700e8578d9aa-home\") pod \"465bc027-ddc4-4f02-9d78-700e8578d9aa\" (UID: \"465bc027-ddc4-4f02-9d78-700e8578d9aa\") " Oct 31 01:26:54.855841 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:54.855677 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/465bc027-ddc4-4f02-9d78-700e8578d9aa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/465bc027-ddc4-4f02-9d78-700e8578d9aa-model-cache\") pod \"465bc027-ddc4-4f02-9d78-700e8578d9aa\" (UID: \"465bc027-ddc4-4f02-9d78-700e8578d9aa\") " Oct 31 01:26:54.855841 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:54.855731 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/465bc027-ddc4-4f02-9d78-700e8578d9aa-kube-api-access-zl2ng\" (UniqueName: \"kubernetes.io/projected/465bc027-ddc4-4f02-9d78-700e8578d9aa-kube-api-access-zl2ng\") pod \"465bc027-ddc4-4f02-9d78-700e8578d9aa\" (UID: \"465bc027-ddc4-4f02-9d78-700e8578d9aa\") " Oct 31 01:26:54.855841 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:54.855780 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/465bc027-ddc4-4f02-9d78-700e8578d9aa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/465bc027-ddc4-4f02-9d78-700e8578d9aa-tmp-dir\") pod \"465bc027-ddc4-4f02-9d78-700e8578d9aa\" (UID: \"465bc027-ddc4-4f02-9d78-700e8578d9aa\") " Oct 31 01:26:54.855841 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:54.855806 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/465bc027-ddc4-4f02-9d78-700e8578d9aa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/465bc027-ddc4-4f02-9d78-700e8578d9aa-dshm\") pod \"465bc027-ddc4-4f02-9d78-700e8578d9aa\" (UID: \"465bc027-ddc4-4f02-9d78-700e8578d9aa\") " Oct 31 01:26:54.856058 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:54.855884 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/465bc027-ddc4-4f02-9d78-700e8578d9aa-home" pod "465bc027-ddc4-4f02-9d78-700e8578d9aa" (UID: "465bc027-ddc4-4f02-9d78-700e8578d9aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:26:54.856058 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:54.855931 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/465bc027-ddc4-4f02-9d78-700e8578d9aa-model-cache" pod "465bc027-ddc4-4f02-9d78-700e8578d9aa" (UID: "465bc027-ddc4-4f02-9d78-700e8578d9aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:26:54.856174 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:54.856073 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/465bc027-ddc4-4f02-9d78-700e8578d9aa-tmp-dir" pod "465bc027-ddc4-4f02-9d78-700e8578d9aa" (UID: "465bc027-ddc4-4f02-9d78-700e8578d9aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:26:54.856319 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:54.856261 4440 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/465bc027-ddc4-4f02-9d78-700e8578d9aa-home\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:26:54.856319 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:54.856283 4440 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/465bc027-ddc4-4f02-9d78-700e8578d9aa-model-cache\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:26:54.856319 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:54.856297 4440 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/465bc027-ddc4-4f02-9d78-700e8578d9aa-tmp-dir\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:26:54.858006 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:54.857975 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/465bc027-ddc4-4f02-9d78-700e8578d9aa-dshm" pod "465bc027-ddc4-4f02-9d78-700e8578d9aa" (UID: "465bc027-ddc4-4f02-9d78-700e8578d9aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:26:54.858116 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:54.858059 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/465bc027-ddc4-4f02-9d78-700e8578d9aa-kube-api-access-zl2ng" pod "465bc027-ddc4-4f02-9d78-700e8578d9aa" (UID: "465bc027-ddc4-4f02-9d78-700e8578d9aa"). InnerVolumeSpecName "kube-api-access-zl2ng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:26:54.858169 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:54.858105 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/465bc027-ddc4-4f02-9d78-700e8578d9aa-tls-certs" pod "465bc027-ddc4-4f02-9d78-700e8578d9aa" (UID: "465bc027-ddc4-4f02-9d78-700e8578d9aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:26:54.957597 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:54.957556 4440 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/465bc027-ddc4-4f02-9d78-700e8578d9aa-tls-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:26:54.957597 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:54.957597 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl2ng\" (UniqueName: \"kubernetes.io/projected/465bc027-ddc4-4f02-9d78-700e8578d9aa-kube-api-access-zl2ng\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:26:54.957868 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:54.957616 4440 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/465bc027-ddc4-4f02-9d78-700e8578d9aa-dshm\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:26:55.037652 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:55.037587 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg" event={"ID":"8762b195-312d-431c-bca2-c9a6dc064f6e","Type":"ContainerDied","Data":"b301064ee3ba4f676f49b6447155cf3ab955e1765d2c8e1dd7e6bc83bde42ca4"} Oct 31 01:26:55.037813 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:55.037677 4440 scope.go:122] "RemoveContainer" containerID="6b80922b2e69b482dc2ec504f3620dfafbaaac836b369b4f2bc377e04bd54a23" Oct 31 01:26:55.038832 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:55.038806 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg" Oct 31 01:26:55.041786 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:55.039951 4440 generic.go:356] "Generic (PLEG): container finished" podID="465bc027-ddc4-4f02-9d78-700e8578d9aa" containerID="77f6b06e36ce74e6d7265f4a645cb19295da9f72448345835845945a4dfff2f6" exitCode=0 Oct 31 01:26:55.041786 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:55.040012 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl" event={"ID":"465bc027-ddc4-4f02-9d78-700e8578d9aa","Type":"ContainerDied","Data":"77f6b06e36ce74e6d7265f4a645cb19295da9f72448345835845945a4dfff2f6"} Oct 31 01:26:55.041786 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:55.040041 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl" event={"ID":"465bc027-ddc4-4f02-9d78-700e8578d9aa","Type":"ContainerDied","Data":"1f424b0eb4b46b43d11ce71ace19c517d705cb671f5bca130285d10535e2493d"} Oct 31 01:26:55.041786 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:55.040103 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl" Oct 31 01:26:55.053312 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:55.053270 4440 scope.go:122] "RemoveContainer" containerID="77f6b06e36ce74e6d7265f4a645cb19295da9f72448345835845945a4dfff2f6" Oct 31 01:26:55.063697 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:55.063661 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg" podUID="8762b195-312d-431c-bca2-c9a6dc064f6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.127:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:26:55.063812 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:55.063762 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg" Oct 31 01:26:55.067046 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:55.067013 4440 scope.go:122] "RemoveContainer" containerID="77f6b06e36ce74e6d7265f4a645cb19295da9f72448345835845945a4dfff2f6" Oct 31 01:26:55.067365 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:26:55.067337 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77f6b06e36ce74e6d7265f4a645cb19295da9f72448345835845945a4dfff2f6\": container with ID starting with 77f6b06e36ce74e6d7265f4a645cb19295da9f72448345835845945a4dfff2f6 not found: ID does not exist" containerID="77f6b06e36ce74e6d7265f4a645cb19295da9f72448345835845945a4dfff2f6" Oct 31 01:26:55.067534 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:55.067378 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77f6b06e36ce74e6d7265f4a645cb19295da9f72448345835845945a4dfff2f6"} err="failed to get container status \"77f6b06e36ce74e6d7265f4a645cb19295da9f72448345835845945a4dfff2f6\": rpc error: code = NotFound desc = could not find container \"77f6b06e36ce74e6d7265f4a645cb19295da9f72448345835845945a4dfff2f6\": container with ID starting with 77f6b06e36ce74e6d7265f4a645cb19295da9f72448345835845945a4dfff2f6 not found: ID does not exist" Oct 31 01:26:55.074164 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:55.074134 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg"] Oct 31 01:26:55.079735 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:55.079705 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-1dd35abe-kserve-856968db7c-jchcg"] Oct 31 01:26:55.094343 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:55.094318 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl"] Oct 31 01:26:55.100269 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:55.100248 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-1dd35abe-kserve-86b86b78b9-tqgkl"] Oct 31 01:26:55.799212 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:55.799175 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="465bc027-ddc4-4f02-9d78-700e8578d9aa" path="/var/lib/kubelet/pods/465bc027-ddc4-4f02-9d78-700e8578d9aa/volumes" Oct 31 01:26:55.800474 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:55.800367 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8762b195-312d-431c-bca2-c9a6dc064f6e" path="/var/lib/kubelet/pods/8762b195-312d-431c-bca2-c9a6dc064f6e/volumes" Oct 31 01:26:56.666494 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:56.666185 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:26:56.923661 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:56.923627 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:26:57.131451 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:57.127685 4440 scope.go:122] "RemoveContainer" containerID="426811f9eb10f6e4c92df4fa04dcf1ae1a9c9b082602ea9184b8c0c7a57f3596" Oct 31 01:26:57.168893 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:26:57.168868 4440 scope.go:122] "RemoveContainer" containerID="51e0fc86ffd801c0edcaae8bc1a9dbb428eeb944031826e32c4fa195b1ad91c1" Oct 31 01:27:01.016392 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:27:01.016344 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:27:12.403549 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:12.403511 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:27:14.047227 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.047193 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-74dcff9898-wvwml"] Oct 31 01:27:14.047689 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.047673 4440 cpu_manager.go:414] "containerMap: removing container" podUID="8762b195-312d-431c-bca2-c9a6dc064f6e" containerName="main" Oct 31 01:27:14.047741 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.047690 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8762b195-312d-431c-bca2-c9a6dc064f6e" containerName="main" Oct 31 01:27:14.047741 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.047700 4440 cpu_manager.go:414] "containerMap: removing container" podUID="2ccb6a7f-bc35-4a1b-b40d-8255cd731f19" containerName="istio-proxy" Oct 31 01:27:14.047741 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.047705 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ccb6a7f-bc35-4a1b-b40d-8255cd731f19" containerName="istio-proxy" Oct 31 01:27:14.047741 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.047715 4440 cpu_manager.go:414] "containerMap: removing container" podUID="465bc027-ddc4-4f02-9d78-700e8578d9aa" containerName="main" Oct 31 01:27:14.047741 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.047720 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="465bc027-ddc4-4f02-9d78-700e8578d9aa" containerName="main" Oct 31 01:27:14.047906 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.047782 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="8762b195-312d-431c-bca2-c9a6dc064f6e" containerName="main" Oct 31 01:27:14.047906 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.047792 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="465bc027-ddc4-4f02-9d78-700e8578d9aa" containerName="main" Oct 31 01:27:14.051378 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.051357 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74dcff9898-wvwml" Oct 31 01:27:14.075819 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.075790 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 01:27:14.075819 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.075797 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 31 01:27:14.076015 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.075823 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-n5p2t\"" Oct 31 01:27:14.077180 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.077166 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:27:14.077309 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.077296 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:27:14.084368 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.084328 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74dcff9898-wvwml"] Oct 31 01:27:14.102110 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.102073 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7fbdfc6954-vfxfr"] Oct 31 01:27:14.106744 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.106714 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7fbdfc6954-vfxfr" Oct 31 01:27:14.127755 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.127723 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7fbdfc6954-vfxfr"] Oct 31 01:27:14.162736 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.162705 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deb65aa2-dd4b-449e-891b-9bfc3b3f3b07-metrics-certs\") pod \"maas-api-7fbdfc6954-vfxfr\" (UID: \"deb65aa2-dd4b-449e-891b-9bfc3b3f3b07\") " pod="odh-ai-gateway-infra/maas-api-7fbdfc6954-vfxfr" Oct 31 01:27:14.162736 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.162737 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq5vm\" (UniqueName: \"kubernetes.io/projected/b2e95efc-837b-4e93-b1c5-16723ce1cb57-kube-api-access-gq5vm\") pod \"maas-api-74dcff9898-wvwml\" (UID: \"b2e95efc-837b-4e93-b1c5-16723ce1cb57\") " pod="odh-ai-gateway-infra/maas-api-74dcff9898-wvwml" Oct 31 01:27:14.162942 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.162780 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deb65aa2-dd4b-449e-891b-9bfc3b3f3b07-maas-api-tls\") pod \"maas-api-7fbdfc6954-vfxfr\" (UID: \"deb65aa2-dd4b-449e-891b-9bfc3b3f3b07\") " pod="odh-ai-gateway-infra/maas-api-7fbdfc6954-vfxfr" Oct 31 01:27:14.162942 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.162826 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2e95efc-837b-4e93-b1c5-16723ce1cb57-maas-api-tls\") pod \"maas-api-74dcff9898-wvwml\" (UID: \"b2e95efc-837b-4e93-b1c5-16723ce1cb57\") " pod="odh-ai-gateway-infra/maas-api-74dcff9898-wvwml" Oct 31 01:27:14.163068 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.162945 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2e95efc-837b-4e93-b1c5-16723ce1cb57-metrics-certs\") pod \"maas-api-74dcff9898-wvwml\" (UID: \"b2e95efc-837b-4e93-b1c5-16723ce1cb57\") " pod="odh-ai-gateway-infra/maas-api-74dcff9898-wvwml" Oct 31 01:27:14.163068 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.162974 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhfvq\" (UniqueName: \"kubernetes.io/projected/deb65aa2-dd4b-449e-891b-9bfc3b3f3b07-kube-api-access-qhfvq\") pod \"maas-api-7fbdfc6954-vfxfr\" (UID: \"deb65aa2-dd4b-449e-891b-9bfc3b3f3b07\") " pod="odh-ai-gateway-infra/maas-api-7fbdfc6954-vfxfr" Oct 31 01:27:14.163068 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.163060 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deb65aa2-dd4b-449e-891b-9bfc3b3f3b07-openshift-service-ca\") pod \"maas-api-7fbdfc6954-vfxfr\" (UID: \"deb65aa2-dd4b-449e-891b-9bfc3b3f3b07\") " pod="odh-ai-gateway-infra/maas-api-7fbdfc6954-vfxfr" Oct 31 01:27:14.163235 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.163084 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2e95efc-837b-4e93-b1c5-16723ce1cb57-openshift-service-ca\") pod \"maas-api-74dcff9898-wvwml\" (UID: \"b2e95efc-837b-4e93-b1c5-16723ce1cb57\") " pod="odh-ai-gateway-infra/maas-api-74dcff9898-wvwml" Oct 31 01:27:14.264080 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.264026 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deb65aa2-dd4b-449e-891b-9bfc3b3f3b07-openshift-service-ca\") pod \"maas-api-7fbdfc6954-vfxfr\" (UID: \"deb65aa2-dd4b-449e-891b-9bfc3b3f3b07\") " pod="odh-ai-gateway-infra/maas-api-7fbdfc6954-vfxfr" Oct 31 01:27:14.264288 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.264089 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2e95efc-837b-4e93-b1c5-16723ce1cb57-openshift-service-ca\") pod \"maas-api-74dcff9898-wvwml\" (UID: \"b2e95efc-837b-4e93-b1c5-16723ce1cb57\") " pod="odh-ai-gateway-infra/maas-api-74dcff9898-wvwml" Oct 31 01:27:14.264288 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.264170 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deb65aa2-dd4b-449e-891b-9bfc3b3f3b07-metrics-certs\") pod \"maas-api-7fbdfc6954-vfxfr\" (UID: \"deb65aa2-dd4b-449e-891b-9bfc3b3f3b07\") " pod="odh-ai-gateway-infra/maas-api-7fbdfc6954-vfxfr" Oct 31 01:27:14.264288 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.264191 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq5vm\" (UniqueName: \"kubernetes.io/projected/b2e95efc-837b-4e93-b1c5-16723ce1cb57-kube-api-access-gq5vm\") pod \"maas-api-74dcff9898-wvwml\" (UID: \"b2e95efc-837b-4e93-b1c5-16723ce1cb57\") " pod="odh-ai-gateway-infra/maas-api-74dcff9898-wvwml" Oct 31 01:27:14.264288 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.264234 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deb65aa2-dd4b-449e-891b-9bfc3b3f3b07-maas-api-tls\") pod \"maas-api-7fbdfc6954-vfxfr\" (UID: \"deb65aa2-dd4b-449e-891b-9bfc3b3f3b07\") " pod="odh-ai-gateway-infra/maas-api-7fbdfc6954-vfxfr" Oct 31 01:27:14.264574 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.264460 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2e95efc-837b-4e93-b1c5-16723ce1cb57-maas-api-tls\") pod \"maas-api-74dcff9898-wvwml\" (UID: \"b2e95efc-837b-4e93-b1c5-16723ce1cb57\") " pod="odh-ai-gateway-infra/maas-api-74dcff9898-wvwml" Oct 31 01:27:14.264574 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.264506 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2e95efc-837b-4e93-b1c5-16723ce1cb57-metrics-certs\") pod \"maas-api-74dcff9898-wvwml\" (UID: \"b2e95efc-837b-4e93-b1c5-16723ce1cb57\") " pod="odh-ai-gateway-infra/maas-api-74dcff9898-wvwml" Oct 31 01:27:14.264574 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.264555 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhfvq\" (UniqueName: \"kubernetes.io/projected/deb65aa2-dd4b-449e-891b-9bfc3b3f3b07-kube-api-access-qhfvq\") pod \"maas-api-7fbdfc6954-vfxfr\" (UID: \"deb65aa2-dd4b-449e-891b-9bfc3b3f3b07\") " pod="odh-ai-gateway-infra/maas-api-7fbdfc6954-vfxfr" Oct 31 01:27:14.265010 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.264986 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deb65aa2-dd4b-449e-891b-9bfc3b3f3b07-openshift-service-ca\") pod \"maas-api-7fbdfc6954-vfxfr\" (UID: \"deb65aa2-dd4b-449e-891b-9bfc3b3f3b07\") " pod="odh-ai-gateway-infra/maas-api-7fbdfc6954-vfxfr" Oct 31 01:27:14.265150 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.264986 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2e95efc-837b-4e93-b1c5-16723ce1cb57-openshift-service-ca\") pod \"maas-api-74dcff9898-wvwml\" (UID: \"b2e95efc-837b-4e93-b1c5-16723ce1cb57\") " pod="odh-ai-gateway-infra/maas-api-74dcff9898-wvwml" Oct 31 01:27:14.267132 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.267109 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deb65aa2-dd4b-449e-891b-9bfc3b3f3b07-maas-api-tls\") pod \"maas-api-7fbdfc6954-vfxfr\" (UID: \"deb65aa2-dd4b-449e-891b-9bfc3b3f3b07\") " pod="odh-ai-gateway-infra/maas-api-7fbdfc6954-vfxfr" Oct 31 01:27:14.267391 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.267372 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2e95efc-837b-4e93-b1c5-16723ce1cb57-metrics-certs\") pod \"maas-api-74dcff9898-wvwml\" (UID: \"b2e95efc-837b-4e93-b1c5-16723ce1cb57\") " pod="odh-ai-gateway-infra/maas-api-74dcff9898-wvwml" Oct 31 01:27:14.267482 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.267403 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deb65aa2-dd4b-449e-891b-9bfc3b3f3b07-metrics-certs\") pod \"maas-api-7fbdfc6954-vfxfr\" (UID: \"deb65aa2-dd4b-449e-891b-9bfc3b3f3b07\") " pod="odh-ai-gateway-infra/maas-api-7fbdfc6954-vfxfr" Oct 31 01:27:14.267885 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.267861 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2e95efc-837b-4e93-b1c5-16723ce1cb57-maas-api-tls\") pod \"maas-api-74dcff9898-wvwml\" (UID: \"b2e95efc-837b-4e93-b1c5-16723ce1cb57\") " pod="odh-ai-gateway-infra/maas-api-74dcff9898-wvwml" Oct 31 01:27:14.275724 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.275705 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq5vm\" (UniqueName: \"kubernetes.io/projected/b2e95efc-837b-4e93-b1c5-16723ce1cb57-kube-api-access-gq5vm\") pod \"maas-api-74dcff9898-wvwml\" (UID: \"b2e95efc-837b-4e93-b1c5-16723ce1cb57\") " pod="odh-ai-gateway-infra/maas-api-74dcff9898-wvwml" Oct 31 01:27:14.276188 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.276164 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhfvq\" (UniqueName: \"kubernetes.io/projected/deb65aa2-dd4b-449e-891b-9bfc3b3f3b07-kube-api-access-qhfvq\") pod \"maas-api-7fbdfc6954-vfxfr\" (UID: \"deb65aa2-dd4b-449e-891b-9bfc3b3f3b07\") " pod="odh-ai-gateway-infra/maas-api-7fbdfc6954-vfxfr" Oct 31 01:27:14.364412 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.364327 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74dcff9898-wvwml" Oct 31 01:27:14.425861 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.425818 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7fbdfc6954-vfxfr" Oct 31 01:27:14.650674 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.650647 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74dcff9898-wvwml"] Oct 31 01:27:14.653615 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:27:14.653586 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e95efc_837b_4e93_b1c5_16723ce1cb57.slice/crio-53bad111a4cb8dbcc96b1f8ac3a5bf842eb67424f3ecf69d8e53cd4d17c7a96c WatchSource:0}: Error finding container 53bad111a4cb8dbcc96b1f8ac3a5bf842eb67424f3ecf69d8e53cd4d17c7a96c: Status 404 returned error can't find the container with id 53bad111a4cb8dbcc96b1f8ac3a5bf842eb67424f3ecf69d8e53cd4d17c7a96c Oct 31 01:27:14.655492 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.655467 4440 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 01:27:14.719606 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:14.719579 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7fbdfc6954-vfxfr"] Oct 31 01:27:14.719951 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:27:14.719926 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeb65aa2_dd4b_449e_891b_9bfc3b3f3b07.slice/crio-f68b7e4febefbb66e31d950a54f5bbe484c5eda8259a735882cc7af26ccc6248 WatchSource:0}: Error finding container f68b7e4febefbb66e31d950a54f5bbe484c5eda8259a735882cc7af26ccc6248: Status 404 returned error can't find the container with id f68b7e4febefbb66e31d950a54f5bbe484c5eda8259a735882cc7af26ccc6248 Oct 31 01:27:15.209969 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:15.209925 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7fbdfc6954-vfxfr" event={"ID":"deb65aa2-dd4b-449e-891b-9bfc3b3f3b07","Type":"ContainerStarted","Data":"f68b7e4febefbb66e31d950a54f5bbe484c5eda8259a735882cc7af26ccc6248"} Oct 31 01:27:15.211346 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:15.211319 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74dcff9898-wvwml" event={"ID":"b2e95efc-837b-4e93-b1c5-16723ce1cb57","Type":"ContainerStarted","Data":"53bad111a4cb8dbcc96b1f8ac3a5bf842eb67424f3ecf69d8e53cd4d17c7a96c"} Oct 31 01:27:16.224876 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:16.224833 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7fbdfc6954-vfxfr" event={"ID":"deb65aa2-dd4b-449e-891b-9bfc3b3f3b07","Type":"ContainerStarted","Data":"4622bce729a0c9546c33d96858971562e10e91eea6da5cc53cc2325107b0168d"} Oct 31 01:27:16.225326 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:16.225030 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7fbdfc6954-vfxfr" Oct 31 01:27:16.226580 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:16.226550 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74dcff9898-wvwml" event={"ID":"b2e95efc-837b-4e93-b1c5-16723ce1cb57","Type":"ContainerStarted","Data":"a4df2a2a2452274128ea907ce17c657cedb505d4e84619d744537a1858296fb0"} Oct 31 01:27:16.226709 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:16.226665 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-74dcff9898-wvwml" Oct 31 01:27:16.250836 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:16.250787 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7fbdfc6954-vfxfr" podStartSLOduration=1.69696064 podStartE2EDuration="2.250776673s" totalImagesPullingTime="553.816033ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:27:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:27:16.247954959 +0000 UTC m=+1880.606677311" watchObservedRunningTime="2026-10-31 01:27:16.250776673 +0000 UTC m=+1880.609498989" Oct 31 01:27:16.272564 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:16.272526 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-74dcff9898-wvwml" podStartSLOduration=1.6489050650000001 podStartE2EDuration="2.272515393s" totalImagesPullingTime="623.610328ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:27:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:27:16.270271807 +0000 UTC m=+1880.628994124" watchObservedRunningTime="2026-10-31 01:27:16.272515393 +0000 UTC m=+1880.631237757" Oct 31 01:27:21.089539 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:21.089502 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:21.836916 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:21.836877 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:22.007762 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:22.007720 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:22.237381 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:22.237352 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7fbdfc6954-vfxfr" Oct 31 01:27:22.237905 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:22.237459 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-74dcff9898-wvwml" Oct 31 01:27:22.329579 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:22.329540 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-74dcff9898-wvwml"] Oct 31 01:27:22.329842 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:22.329809 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-74dcff9898-wvwml" podUID="b2e95efc-837b-4e93-b1c5-16723ce1cb57" containerName="maas-api" containerID="cri-o://a4df2a2a2452274128ea907ce17c657cedb505d4e84619d744537a1858296fb0" gracePeriod=30 Oct 31 01:27:22.578965 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:22.578940 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74dcff9898-wvwml" Oct 31 01:27:22.644152 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:22.644119 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2e95efc-837b-4e93-b1c5-16723ce1cb57-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2e95efc-837b-4e93-b1c5-16723ce1cb57-maas-api-tls\") pod \"b2e95efc-837b-4e93-b1c5-16723ce1cb57\" (UID: \"b2e95efc-837b-4e93-b1c5-16723ce1cb57\") " Oct 31 01:27:22.644312 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:22.644174 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2e95efc-837b-4e93-b1c5-16723ce1cb57-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2e95efc-837b-4e93-b1c5-16723ce1cb57-openshift-service-ca\") pod \"b2e95efc-837b-4e93-b1c5-16723ce1cb57\" (UID: \"b2e95efc-837b-4e93-b1c5-16723ce1cb57\") " Oct 31 01:27:22.644312 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:22.644196 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2e95efc-837b-4e93-b1c5-16723ce1cb57-kube-api-access-gq5vm\" (UniqueName: \"kubernetes.io/projected/b2e95efc-837b-4e93-b1c5-16723ce1cb57-kube-api-access-gq5vm\") pod \"b2e95efc-837b-4e93-b1c5-16723ce1cb57\" (UID: \"b2e95efc-837b-4e93-b1c5-16723ce1cb57\") " Oct 31 01:27:22.644312 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:22.644250 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2e95efc-837b-4e93-b1c5-16723ce1cb57-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2e95efc-837b-4e93-b1c5-16723ce1cb57-metrics-certs\") pod \"b2e95efc-837b-4e93-b1c5-16723ce1cb57\" (UID: \"b2e95efc-837b-4e93-b1c5-16723ce1cb57\") " Oct 31 01:27:22.644593 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:22.644565 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2e95efc-837b-4e93-b1c5-16723ce1cb57-openshift-service-ca" pod "b2e95efc-837b-4e93-b1c5-16723ce1cb57" (UID: "b2e95efc-837b-4e93-b1c5-16723ce1cb57"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:27:22.646330 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:22.646301 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2e95efc-837b-4e93-b1c5-16723ce1cb57-kube-api-access-gq5vm" pod "b2e95efc-837b-4e93-b1c5-16723ce1cb57" (UID: "b2e95efc-837b-4e93-b1c5-16723ce1cb57"). InnerVolumeSpecName "kube-api-access-gq5vm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:27:22.646478 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:22.646396 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2e95efc-837b-4e93-b1c5-16723ce1cb57-maas-api-tls" pod "b2e95efc-837b-4e93-b1c5-16723ce1cb57" (UID: "b2e95efc-837b-4e93-b1c5-16723ce1cb57"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:27:22.646478 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:22.646409 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2e95efc-837b-4e93-b1c5-16723ce1cb57-metrics-certs" pod "b2e95efc-837b-4e93-b1c5-16723ce1cb57" (UID: "b2e95efc-837b-4e93-b1c5-16723ce1cb57"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:27:22.746086 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:22.746048 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2e95efc-837b-4e93-b1c5-16723ce1cb57-maas-api-tls\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:22.746086 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:22.746084 4440 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2e95efc-837b-4e93-b1c5-16723ce1cb57-openshift-service-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:22.746265 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:22.746099 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq5vm\" (UniqueName: \"kubernetes.io/projected/b2e95efc-837b-4e93-b1c5-16723ce1cb57-kube-api-access-gq5vm\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:22.746265 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:22.746114 4440 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2e95efc-837b-4e93-b1c5-16723ce1cb57-metrics-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:23.266934 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:23.266899 4440 generic.go:356] "Generic (PLEG): container finished" podID="b2e95efc-837b-4e93-b1c5-16723ce1cb57" containerID="a4df2a2a2452274128ea907ce17c657cedb505d4e84619d744537a1858296fb0" exitCode=0 Oct 31 01:27:23.267388 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:23.266953 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74dcff9898-wvwml" Oct 31 01:27:23.267388 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:23.266986 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74dcff9898-wvwml" event={"ID":"b2e95efc-837b-4e93-b1c5-16723ce1cb57","Type":"ContainerDied","Data":"a4df2a2a2452274128ea907ce17c657cedb505d4e84619d744537a1858296fb0"} Oct 31 01:27:23.267388 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:23.267052 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74dcff9898-wvwml" event={"ID":"b2e95efc-837b-4e93-b1c5-16723ce1cb57","Type":"ContainerDied","Data":"53bad111a4cb8dbcc96b1f8ac3a5bf842eb67424f3ecf69d8e53cd4d17c7a96c"} Oct 31 01:27:23.267388 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:23.267083 4440 scope.go:122] "RemoveContainer" containerID="a4df2a2a2452274128ea907ce17c657cedb505d4e84619d744537a1858296fb0" Oct 31 01:27:23.277363 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:23.277341 4440 scope.go:122] "RemoveContainer" containerID="a4df2a2a2452274128ea907ce17c657cedb505d4e84619d744537a1858296fb0" Oct 31 01:27:23.277655 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:27:23.277636 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4df2a2a2452274128ea907ce17c657cedb505d4e84619d744537a1858296fb0\": container with ID starting with a4df2a2a2452274128ea907ce17c657cedb505d4e84619d744537a1858296fb0 not found: ID does not exist" containerID="a4df2a2a2452274128ea907ce17c657cedb505d4e84619d744537a1858296fb0" Oct 31 01:27:23.277751 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:23.277663 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4df2a2a2452274128ea907ce17c657cedb505d4e84619d744537a1858296fb0"} err="failed to get container status \"a4df2a2a2452274128ea907ce17c657cedb505d4e84619d744537a1858296fb0\": rpc error: code = NotFound desc = could not find container \"a4df2a2a2452274128ea907ce17c657cedb505d4e84619d744537a1858296fb0\": container with ID starting with a4df2a2a2452274128ea907ce17c657cedb505d4e84619d744537a1858296fb0 not found: ID does not exist" Oct 31 01:27:23.290149 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:23.290127 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-74dcff9898-wvwml"] Oct 31 01:27:23.296935 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:23.296912 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-74dcff9898-wvwml"] Oct 31 01:27:23.769809 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:23.769775 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2e95efc-837b-4e93-b1c5-16723ce1cb57" path="/var/lib/kubelet/pods/b2e95efc-837b-4e93-b1c5-16723ce1cb57/volumes" Oct 31 01:27:24.481556 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:24.481402 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:24.693050 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:24.693014 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:25.632843 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:25.632802 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:25.788745 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:25.788709 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b68f43ca-55c9549474-gf7jx"] Oct 31 01:27:25.789284 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:25.789259 4440 cpu_manager.go:414] "containerMap: removing container" podUID="b2e95efc-837b-4e93-b1c5-16723ce1cb57" containerName="maas-api" Oct 31 01:27:25.789284 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:25.789284 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2e95efc-837b-4e93-b1c5-16723ce1cb57" containerName="maas-api" Oct 31 01:27:25.789496 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:25.789375 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2e95efc-837b-4e93-b1c5-16723ce1cb57" containerName="maas-api" Oct 31 01:27:25.792002 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:25.791953 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:25.793984 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:25.793961 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b68f43ca-55c9549474-gf7jx"] Oct 31 01:27:25.794089 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:25.794062 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b68f43ca-55c9549474-gf7jx" Oct 31 01:27:25.810183 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:25.810160 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b68f43ca-serving-cert\"" Oct 31 01:27:25.904916 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:25.904809 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smtqd\" (UniqueName: \"kubernetes.io/projected/1fc8dcf0-8358-49f6-bc52-6c33ea6837e1-kube-api-access-smtqd\") pod \"maas-api-e2e-ait-b68f43ca-55c9549474-gf7jx\" (UID: \"1fc8dcf0-8358-49f6-bc52-6c33ea6837e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b68f43ca-55c9549474-gf7jx" Oct 31 01:27:25.904916 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:25.904908 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fc8dcf0-8358-49f6-bc52-6c33ea6837e1-maas-api-tls\") pod \"maas-api-e2e-ait-b68f43ca-55c9549474-gf7jx\" (UID: \"1fc8dcf0-8358-49f6-bc52-6c33ea6837e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b68f43ca-55c9549474-gf7jx" Oct 31 01:27:25.905153 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:25.904935 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fc8dcf0-8358-49f6-bc52-6c33ea6837e1-openshift-service-ca\") pod \"maas-api-e2e-ait-b68f43ca-55c9549474-gf7jx\" (UID: \"1fc8dcf0-8358-49f6-bc52-6c33ea6837e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b68f43ca-55c9549474-gf7jx" Oct 31 01:27:25.905153 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:25.904986 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fc8dcf0-8358-49f6-bc52-6c33ea6837e1-metrics-certs\") pod \"maas-api-e2e-ait-b68f43ca-55c9549474-gf7jx\" (UID: \"1fc8dcf0-8358-49f6-bc52-6c33ea6837e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b68f43ca-55c9549474-gf7jx" Oct 31 01:27:25.918809 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:25.918767 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:26.006236 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:26.006200 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fc8dcf0-8358-49f6-bc52-6c33ea6837e1-metrics-certs\") pod \"maas-api-e2e-ait-b68f43ca-55c9549474-gf7jx\" (UID: \"1fc8dcf0-8358-49f6-bc52-6c33ea6837e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b68f43ca-55c9549474-gf7jx" Oct 31 01:27:26.006401 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:26.006308 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smtqd\" (UniqueName: \"kubernetes.io/projected/1fc8dcf0-8358-49f6-bc52-6c33ea6837e1-kube-api-access-smtqd\") pod \"maas-api-e2e-ait-b68f43ca-55c9549474-gf7jx\" (UID: \"1fc8dcf0-8358-49f6-bc52-6c33ea6837e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b68f43ca-55c9549474-gf7jx" Oct 31 01:27:26.006401 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:26.006349 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fc8dcf0-8358-49f6-bc52-6c33ea6837e1-maas-api-tls\") pod \"maas-api-e2e-ait-b68f43ca-55c9549474-gf7jx\" (UID: \"1fc8dcf0-8358-49f6-bc52-6c33ea6837e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b68f43ca-55c9549474-gf7jx" Oct 31 01:27:26.006401 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:26.006366 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fc8dcf0-8358-49f6-bc52-6c33ea6837e1-openshift-service-ca\") pod \"maas-api-e2e-ait-b68f43ca-55c9549474-gf7jx\" (UID: \"1fc8dcf0-8358-49f6-bc52-6c33ea6837e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b68f43ca-55c9549474-gf7jx" Oct 31 01:27:26.007043 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:26.007019 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fc8dcf0-8358-49f6-bc52-6c33ea6837e1-openshift-service-ca\") pod \"maas-api-e2e-ait-b68f43ca-55c9549474-gf7jx\" (UID: \"1fc8dcf0-8358-49f6-bc52-6c33ea6837e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b68f43ca-55c9549474-gf7jx" Oct 31 01:27:26.008765 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:26.008743 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fc8dcf0-8358-49f6-bc52-6c33ea6837e1-metrics-certs\") pod \"maas-api-e2e-ait-b68f43ca-55c9549474-gf7jx\" (UID: \"1fc8dcf0-8358-49f6-bc52-6c33ea6837e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b68f43ca-55c9549474-gf7jx" Oct 31 01:27:26.008853 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:26.008792 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fc8dcf0-8358-49f6-bc52-6c33ea6837e1-maas-api-tls\") pod \"maas-api-e2e-ait-b68f43ca-55c9549474-gf7jx\" (UID: \"1fc8dcf0-8358-49f6-bc52-6c33ea6837e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b68f43ca-55c9549474-gf7jx" Oct 31 01:27:26.014791 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:26.014771 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smtqd\" (UniqueName: \"kubernetes.io/projected/1fc8dcf0-8358-49f6-bc52-6c33ea6837e1-kube-api-access-smtqd\") pod \"maas-api-e2e-ait-b68f43ca-55c9549474-gf7jx\" (UID: \"1fc8dcf0-8358-49f6-bc52-6c33ea6837e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b68f43ca-55c9549474-gf7jx" Oct 31 01:27:26.054572 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:26.054544 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:26.109583 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:26.109559 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b68f43ca-55c9549474-gf7jx" Oct 31 01:27:26.398799 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:26.398775 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b68f43ca-55c9549474-gf7jx"] Oct 31 01:27:26.399858 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:27:26.399832 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fc8dcf0_8358_49f6_bc52_6c33ea6837e1.slice/crio-162cafec79ab5994ed6c40beebc79e11c120847be88d0293b2a2ca83cfc028fb WatchSource:0}: Error finding container 162cafec79ab5994ed6c40beebc79e11c120847be88d0293b2a2ca83cfc028fb: Status 404 returned error can't find the container with id 162cafec79ab5994ed6c40beebc79e11c120847be88d0293b2a2ca83cfc028fb Oct 31 01:27:26.856579 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:26.856547 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:27.291272 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:27.291239 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b68f43ca-55c9549474-gf7jx" event={"ID":"1fc8dcf0-8358-49f6-bc52-6c33ea6837e1","Type":"ContainerStarted","Data":"3d629c217ab5ce4981ec296b57ab5fe253ca2fa6bce970af9e01828683c46d46"} Oct 31 01:27:27.291481 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:27.291287 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b68f43ca-55c9549474-gf7jx" event={"ID":"1fc8dcf0-8358-49f6-bc52-6c33ea6837e1","Type":"ContainerStarted","Data":"162cafec79ab5994ed6c40beebc79e11c120847be88d0293b2a2ca83cfc028fb"} Oct 31 01:27:27.291481 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:27.291448 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b68f43ca-55c9549474-gf7jx" Oct 31 01:27:27.316013 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:27.315954 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b68f43ca-55c9549474-gf7jx" podStartSLOduration=1.890043718 podStartE2EDuration="2.315938656s" totalImagesPullingTime="425.894938ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:27:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:27:27.312277481 +0000 UTC m=+1891.670999797" watchObservedRunningTime="2026-10-31 01:27:27.315938656 +0000 UTC m=+1891.674660976" Oct 31 01:27:27.621177 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:27.621073 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:28.515844 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:28.515804 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:28.683895 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:28.683853 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:27:29.378308 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:29.378272 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:29.983276 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:29.983235 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:30.697112 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:30.697079 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:31.006156 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:27:31.006067 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:27:31.285292 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:31.285194 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:32.003410 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:32.003372 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:32.691647 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:32.691610 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:33.303252 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:33.303220 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b68f43ca-55c9549474-gf7jx" Oct 31 01:27:33.401160 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:33.401126 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:34.139630 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:34.139595 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:35.205213 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:35.205183 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:35.911507 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:35.911467 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:36.591431 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:36.591385 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:37.312462 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:37.312410 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:38.007133 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:38.007074 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:38.676528 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:38.676491 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:39.389268 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:39.389232 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:40.083498 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:40.083463 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:41.037385 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:41.037350 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:41.910536 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:41.910505 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:42.709026 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:42.708991 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:43.450029 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:43.449997 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:44.115520 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:44.115470 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:44.807469 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:44.807408 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:45.509193 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:45.509152 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:46.311373 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:46.311335 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:47.009817 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:47.009779 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:47.306196 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:47.306113 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:47.734275 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:47.734218 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:48.498838 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:48.498804 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:49.250393 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:49.250361 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:49.905873 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:49.905838 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:50.539284 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:50.539245 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:50.674054 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:50.674021 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:51.107314 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:51.107279 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b68f43ca-55c9549474-gf7jx"] Oct 31 01:27:51.108750 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:51.108718 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b68f43ca-55c9549474-gf7jx" podUID="1fc8dcf0-8358-49f6-bc52-6c33ea6837e1" containerName="maas-api" containerID="cri-o://3d629c217ab5ce4981ec296b57ab5fe253ca2fa6bce970af9e01828683c46d46" gracePeriod=30 Oct 31 01:27:51.223939 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:51.223902 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:51.361841 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:51.361756 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:51.389515 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:51.389494 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b68f43ca-55c9549474-gf7jx" Oct 31 01:27:51.439679 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:51.439642 4440 generic.go:356] "Generic (PLEG): container finished" podID="1fc8dcf0-8358-49f6-bc52-6c33ea6837e1" containerID="3d629c217ab5ce4981ec296b57ab5fe253ca2fa6bce970af9e01828683c46d46" exitCode=0 Oct 31 01:27:51.439927 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:51.439702 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b68f43ca-55c9549474-gf7jx" Oct 31 01:27:51.439927 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:51.439682 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b68f43ca-55c9549474-gf7jx" event={"ID":"1fc8dcf0-8358-49f6-bc52-6c33ea6837e1","Type":"ContainerDied","Data":"3d629c217ab5ce4981ec296b57ab5fe253ca2fa6bce970af9e01828683c46d46"} Oct 31 01:27:51.439927 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:51.439840 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b68f43ca-55c9549474-gf7jx" event={"ID":"1fc8dcf0-8358-49f6-bc52-6c33ea6837e1","Type":"ContainerDied","Data":"162cafec79ab5994ed6c40beebc79e11c120847be88d0293b2a2ca83cfc028fb"} Oct 31 01:27:51.439927 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:51.439873 4440 scope.go:122] "RemoveContainer" containerID="3d629c217ab5ce4981ec296b57ab5fe253ca2fa6bce970af9e01828683c46d46" Oct 31 01:27:51.450188 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:51.450172 4440 scope.go:122] "RemoveContainer" containerID="3d629c217ab5ce4981ec296b57ab5fe253ca2fa6bce970af9e01828683c46d46" Oct 31 01:27:51.450493 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:27:51.450472 4440 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d629c217ab5ce4981ec296b57ab5fe253ca2fa6bce970af9e01828683c46d46\": container with ID starting with 3d629c217ab5ce4981ec296b57ab5fe253ca2fa6bce970af9e01828683c46d46 not found: ID does not exist" containerID="3d629c217ab5ce4981ec296b57ab5fe253ca2fa6bce970af9e01828683c46d46" Oct 31 01:27:51.450576 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:51.450513 4440 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d629c217ab5ce4981ec296b57ab5fe253ca2fa6bce970af9e01828683c46d46"} err="failed to get container status \"3d629c217ab5ce4981ec296b57ab5fe253ca2fa6bce970af9e01828683c46d46\": rpc error: code = NotFound desc = could not find container \"3d629c217ab5ce4981ec296b57ab5fe253ca2fa6bce970af9e01828683c46d46\": container with ID starting with 3d629c217ab5ce4981ec296b57ab5fe253ca2fa6bce970af9e01828683c46d46 not found: ID does not exist" Oct 31 01:27:51.469437 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:51.469397 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fc8dcf0-8358-49f6-bc52-6c33ea6837e1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fc8dcf0-8358-49f6-bc52-6c33ea6837e1-metrics-certs\") pod \"1fc8dcf0-8358-49f6-bc52-6c33ea6837e1\" (UID: \"1fc8dcf0-8358-49f6-bc52-6c33ea6837e1\") " Oct 31 01:27:51.469560 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:51.469484 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fc8dcf0-8358-49f6-bc52-6c33ea6837e1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fc8dcf0-8358-49f6-bc52-6c33ea6837e1-openshift-service-ca\") pod \"1fc8dcf0-8358-49f6-bc52-6c33ea6837e1\" (UID: \"1fc8dcf0-8358-49f6-bc52-6c33ea6837e1\") " Oct 31 01:27:51.469560 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:51.469514 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fc8dcf0-8358-49f6-bc52-6c33ea6837e1-kube-api-access-smtqd\" (UniqueName: \"kubernetes.io/projected/1fc8dcf0-8358-49f6-bc52-6c33ea6837e1-kube-api-access-smtqd\") pod \"1fc8dcf0-8358-49f6-bc52-6c33ea6837e1\" (UID: \"1fc8dcf0-8358-49f6-bc52-6c33ea6837e1\") " Oct 31 01:27:51.469675 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:51.469559 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fc8dcf0-8358-49f6-bc52-6c33ea6837e1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fc8dcf0-8358-49f6-bc52-6c33ea6837e1-maas-api-tls\") pod \"1fc8dcf0-8358-49f6-bc52-6c33ea6837e1\" (UID: \"1fc8dcf0-8358-49f6-bc52-6c33ea6837e1\") " Oct 31 01:27:51.469890 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:51.469866 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fc8dcf0-8358-49f6-bc52-6c33ea6837e1-openshift-service-ca" pod "1fc8dcf0-8358-49f6-bc52-6c33ea6837e1" (UID: "1fc8dcf0-8358-49f6-bc52-6c33ea6837e1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:27:51.471868 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:51.471829 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fc8dcf0-8358-49f6-bc52-6c33ea6837e1-kube-api-access-smtqd" pod "1fc8dcf0-8358-49f6-bc52-6c33ea6837e1" (UID: "1fc8dcf0-8358-49f6-bc52-6c33ea6837e1"). InnerVolumeSpecName "kube-api-access-smtqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:27:51.471987 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:51.471885 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fc8dcf0-8358-49f6-bc52-6c33ea6837e1-metrics-certs" pod "1fc8dcf0-8358-49f6-bc52-6c33ea6837e1" (UID: "1fc8dcf0-8358-49f6-bc52-6c33ea6837e1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:27:51.471987 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:51.471910 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fc8dcf0-8358-49f6-bc52-6c33ea6837e1-maas-api-tls" pod "1fc8dcf0-8358-49f6-bc52-6c33ea6837e1" (UID: "1fc8dcf0-8358-49f6-bc52-6c33ea6837e1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:27:51.492521 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:51.492493 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:51.570688 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:51.570653 4440 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fc8dcf0-8358-49f6-bc52-6c33ea6837e1-metrics-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:51.570688 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:51.570683 4440 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fc8dcf0-8358-49f6-bc52-6c33ea6837e1-openshift-service-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:51.570688 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:51.570692 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smtqd\" (UniqueName: \"kubernetes.io/projected/1fc8dcf0-8358-49f6-bc52-6c33ea6837e1-kube-api-access-smtqd\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:51.570931 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:51.570701 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fc8dcf0-8358-49f6-bc52-6c33ea6837e1-maas-api-tls\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:51.771026 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:51.771001 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b68f43ca-55c9549474-gf7jx"] Oct 31 01:27:51.771271 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:51.771040 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b68f43ca-55c9549474-gf7jx"] Oct 31 01:27:52.817316 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:52.817276 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:27:53.770121 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:53.770086 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fc8dcf0-8358-49f6-bc52-6c33ea6837e1" path="/var/lib/kubelet/pods/1fc8dcf0-8358-49f6-bc52-6c33ea6837e1/volumes" Oct 31 01:27:56.756221 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:56.756189 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:58.046772 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:58.046737 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:58.530049 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:58.530014 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:58.671634 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:27:58.671588 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:00.999505 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:28:00.999464 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:28:02.353535 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:02.353496 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:02.468305 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:02.468256 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:03.413400 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:03.413365 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cec774-74f45f5789-2nbrx"] Oct 31 01:28:03.413895 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:03.413820 4440 cpu_manager.go:414] "containerMap: removing container" podUID="1fc8dcf0-8358-49f6-bc52-6c33ea6837e1" containerName="maas-api" Oct 31 01:28:03.413895 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:03.413833 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fc8dcf0-8358-49f6-bc52-6c33ea6837e1" containerName="maas-api" Oct 31 01:28:03.414018 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:03.413935 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fc8dcf0-8358-49f6-bc52-6c33ea6837e1" containerName="maas-api" Oct 31 01:28:03.418846 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:03.418825 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cec774-74f45f5789-2nbrx" Oct 31 01:28:03.433148 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:03.433104 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-cec774-serving-cert\"" Oct 31 01:28:03.436738 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:03.436571 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cec774-74f45f5789-2nbrx"] Oct 31 01:28:03.587828 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:03.587795 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/747609de-e694-475a-803a-f759ee56c30c-metrics-certs\") pod \"maas-api-e2e-authcfg-http-cec774-74f45f5789-2nbrx\" (UID: \"747609de-e694-475a-803a-f759ee56c30c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cec774-74f45f5789-2nbrx" Oct 31 01:28:03.587993 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:03.587863 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqcd2\" (UniqueName: \"kubernetes.io/projected/747609de-e694-475a-803a-f759ee56c30c-kube-api-access-pqcd2\") pod \"maas-api-e2e-authcfg-http-cec774-74f45f5789-2nbrx\" (UID: \"747609de-e694-475a-803a-f759ee56c30c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cec774-74f45f5789-2nbrx" Oct 31 01:28:03.587993 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:03.587980 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/747609de-e694-475a-803a-f759ee56c30c-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-cec774-74f45f5789-2nbrx\" (UID: \"747609de-e694-475a-803a-f759ee56c30c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cec774-74f45f5789-2nbrx" Oct 31 01:28:03.588073 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:03.588014 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/747609de-e694-475a-803a-f759ee56c30c-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-cec774-74f45f5789-2nbrx\" (UID: \"747609de-e694-475a-803a-f759ee56c30c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cec774-74f45f5789-2nbrx" Oct 31 01:28:03.605707 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:03.605671 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:03.689079 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:03.689044 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/747609de-e694-475a-803a-f759ee56c30c-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-cec774-74f45f5789-2nbrx\" (UID: \"747609de-e694-475a-803a-f759ee56c30c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cec774-74f45f5789-2nbrx" Oct 31 01:28:03.689250 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:03.689111 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/747609de-e694-475a-803a-f759ee56c30c-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-cec774-74f45f5789-2nbrx\" (UID: \"747609de-e694-475a-803a-f759ee56c30c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cec774-74f45f5789-2nbrx" Oct 31 01:28:03.689250 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:03.689161 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/747609de-e694-475a-803a-f759ee56c30c-metrics-certs\") pod \"maas-api-e2e-authcfg-http-cec774-74f45f5789-2nbrx\" (UID: \"747609de-e694-475a-803a-f759ee56c30c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cec774-74f45f5789-2nbrx" Oct 31 01:28:03.689250 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:03.689232 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqcd2\" (UniqueName: \"kubernetes.io/projected/747609de-e694-475a-803a-f759ee56c30c-kube-api-access-pqcd2\") pod \"maas-api-e2e-authcfg-http-cec774-74f45f5789-2nbrx\" (UID: \"747609de-e694-475a-803a-f759ee56c30c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cec774-74f45f5789-2nbrx" Oct 31 01:28:03.689852 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:03.689829 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/747609de-e694-475a-803a-f759ee56c30c-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-cec774-74f45f5789-2nbrx\" (UID: \"747609de-e694-475a-803a-f759ee56c30c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cec774-74f45f5789-2nbrx" Oct 31 01:28:03.691765 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:03.691736 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/747609de-e694-475a-803a-f759ee56c30c-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-cec774-74f45f5789-2nbrx\" (UID: \"747609de-e694-475a-803a-f759ee56c30c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cec774-74f45f5789-2nbrx" Oct 31 01:28:03.691898 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:03.691818 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/747609de-e694-475a-803a-f759ee56c30c-metrics-certs\") pod \"maas-api-e2e-authcfg-http-cec774-74f45f5789-2nbrx\" (UID: \"747609de-e694-475a-803a-f759ee56c30c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cec774-74f45f5789-2nbrx" Oct 31 01:28:03.698928 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:03.698902 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqcd2\" (UniqueName: \"kubernetes.io/projected/747609de-e694-475a-803a-f759ee56c30c-kube-api-access-pqcd2\") pod \"maas-api-e2e-authcfg-http-cec774-74f45f5789-2nbrx\" (UID: \"747609de-e694-475a-803a-f759ee56c30c\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cec774-74f45f5789-2nbrx" Oct 31 01:28:03.731691 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:03.731651 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:03.735559 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:03.734267 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cec774-74f45f5789-2nbrx" Oct 31 01:28:04.040946 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:04.040918 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cec774-74f45f5789-2nbrx"] Oct 31 01:28:04.042353 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:28:04.042320 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod747609de_e694_475a_803a_f759ee56c30c.slice/crio-ba6118326f3c5c445df4a6bab58448c49d42aebb4bf56d08a63e7aed35ebbc7d WatchSource:0}: Error finding container ba6118326f3c5c445df4a6bab58448c49d42aebb4bf56d08a63e7aed35ebbc7d: Status 404 returned error can't find the container with id ba6118326f3c5c445df4a6bab58448c49d42aebb4bf56d08a63e7aed35ebbc7d Oct 31 01:28:04.480027 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:04.479990 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:04.525732 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:04.525700 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cec774-74f45f5789-2nbrx" event={"ID":"747609de-e694-475a-803a-f759ee56c30c","Type":"ContainerStarted","Data":"ba6118326f3c5c445df4a6bab58448c49d42aebb4bf56d08a63e7aed35ebbc7d"} Oct 31 01:28:05.196998 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:05.196947 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:05.535052 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:05.534943 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cec774-74f45f5789-2nbrx" event={"ID":"747609de-e694-475a-803a-f759ee56c30c","Type":"ContainerStarted","Data":"a4201c91b2fa39993637106a7f88d5c1321524d23052a630755c83191e034a8a"} Oct 31 01:28:05.535052 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:05.535048 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cec774-74f45f5789-2nbrx" Oct 31 01:28:06.008919 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:06.008873 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:06.221425 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:06.221364 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cec774-74f45f5789-2nbrx" podStartSLOduration=2.769036379 podStartE2EDuration="3.221342564s" totalImagesPullingTime="452.306185ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:28:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:28:05.55699778 +0000 UTC m=+1929.915720099" watchObservedRunningTime="2026-10-31 01:28:06.221342564 +0000 UTC m=+1930.580064881" Oct 31 01:28:06.223498 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:06.223475 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:28:07.142516 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:07.142411 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:07.909045 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:07.909003 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:08.630340 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:08.630268 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:09.506180 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:09.506134 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:10.225437 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:10.224113 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:10.880627 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:10.880593 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:11.547446 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:11.547384 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:11.552654 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:11.552632 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cec774-74f45f5789-2nbrx" Oct 31 01:28:12.247334 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:12.247292 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:12.986353 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:12.986318 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:13.701028 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:13.700985 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:14.294562 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:14.294525 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:14.898983 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:14.898949 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:15.588469 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:15.588413 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:16.278461 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:16.278400 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:16.912795 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:16.912749 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:17.344408 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:17.344374 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:17.607774 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:17.607686 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:18.351612 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:18.351578 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:18.997253 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:18.997214 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:19.859414 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:19.859377 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:20.695498 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:20.695458 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:21.294974 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:21.294925 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:21.920977 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:21.920943 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:22.608713 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:22.608678 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:23.280405 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:23.280372 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:24.101297 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:24.101256 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:24.794959 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:24.794922 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:25.488731 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:25.488698 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:26.238429 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:26.238381 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:26.894672 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:26.894630 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:27.581948 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:27.581910 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:28.206137 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:28.206103 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:28.819271 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:28.819238 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:29.483832 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:29.483800 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:30.096994 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:30.096955 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:30.769764 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:30.769716 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:30.999859 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:28:30.999821 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:28:31.384831 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:31.384798 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:32.086796 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:32.086760 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:32.692352 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:32.692305 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:33.293406 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:33.293366 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:34.005858 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:34.005821 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:34.699574 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:34.699538 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:35.297320 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:35.297282 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:35.946108 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:35.946070 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:36.600842 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:36.600806 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:37.275480 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:37.275440 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:37.954453 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:37.954403 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:38.693103 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:38.693062 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:39.407326 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:39.407279 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:40.094874 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:40.094839 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:40.798847 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:40.798811 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:41.475320 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:41.475290 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:42.083830 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:42.083791 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:42.822962 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:42.822927 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:43.468026 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:43.467984 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:44.170981 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:44.170945 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:44.706126 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:44.706088 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:45.392730 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:45.392695 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:46.226396 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:46.226348 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:46.358154 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:46.358116 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:46.891705 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:46.891669 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:47.336287 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:47.336235 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:47.594495 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:47.594385 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:48.282124 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:48.282090 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:48.968536 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:48.968493 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:49.652200 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:49.652162 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:50.296833 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:50.296802 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:50.887298 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:50.887258 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:51.573788 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:51.573748 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:52.292131 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:52.292095 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:52.896903 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:52.896866 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:53.491242 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:53.491205 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:54.206030 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:54.205979 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:54.921906 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:54.921871 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:55.582618 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:55.582565 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:56.295709 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:56.295667 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:56.901329 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:56.901291 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:57.695129 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:57.695094 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:58.303001 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:58.302965 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:58.976466 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:58.976429 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:59.601795 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:28:59.601762 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:00.284263 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:00.284230 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:00.529674 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:00.529634 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:00.986467 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:00.986402 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:01.015864 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:29:01.015820 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:29:01.653950 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:01.653904 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:02.288673 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:02.288642 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:02.898412 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:02.898375 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:03.577975 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:03.577939 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:04.272193 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:04.272156 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:04.894542 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:04.894506 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:05.564635 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:05.564584 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:06.202918 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:06.202869 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:06.886887 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:06.886842 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:07.502616 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:07.502579 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:08.183790 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:08.183755 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:08.920109 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:08.920070 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:09.581764 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:09.581734 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:10.280958 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:10.280921 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:10.886201 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:10.886160 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:11.497112 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:11.497077 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:12.181257 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:12.181220 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:12.808298 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:12.808252 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:13.461929 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:13.461891 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:14.089235 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:14.089195 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:14.691742 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:14.691691 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:15.498169 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:15.498135 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:16.207079 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:16.207046 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:17.017295 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:17.017261 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:17.712467 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:17.712430 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:18.398780 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:18.398740 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:19.082002 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:19.081964 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:19.517601 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:19.517565 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:19.786982 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:19.786892 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:20.491341 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:20.491306 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:21.194707 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:21.194673 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:21.877485 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:21.877451 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:22.497741 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:22.497709 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:23.212293 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:23.212252 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:23.876047 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:23.876008 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:24.495653 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:24.495612 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:25.170005 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:25.169968 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:25.777224 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:25.777189 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:26.052547 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:26.052451 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:26.525854 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:26.525824 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:27.097824 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:27.097789 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:27.718059 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:27.718028 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:28.389739 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:28.389701 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:29.105117 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:29.105084 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:29.691171 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:29.691129 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:30.304910 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:30.304877 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:30.999412 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:29:30.999374 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:29:31.004681 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:31.004642 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:31.679986 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:31.679955 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:32.312168 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:32.312129 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:32.989435 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:32.989386 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:33.592957 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:33.592924 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:34.446193 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:34.446154 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:34.985556 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:34.985525 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:35.632899 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:35.632862 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:36.341066 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:36.341026 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:36.898310 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:36.898274 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:37.498239 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:37.498203 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:38.377530 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:38.377494 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:39.024115 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:39.024083 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:39.715949 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:39.715915 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:40.389150 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:40.389110 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:41.094170 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:41.094132 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:41.693281 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:41.693247 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:42.297259 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:42.297219 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:42.979401 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:42.979366 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:43.597779 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:43.597749 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:44.270608 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:44.270563 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:44.894022 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:44.893975 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:45.589680 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:45.589649 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:46.203148 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:46.203117 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:46.801167 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:46.801109 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:47.383267 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:47.383231 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:47.496168 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:47.496127 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:48.001651 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:48.001617 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:48.106823 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:48.106780 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:52.934984 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:52.934944 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cec774-74f45f5789-2nbrx"] Oct 31 01:29:52.935796 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:52.935198 4440 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cec774-74f45f5789-2nbrx" podUID="747609de-e694-475a-803a-f759ee56c30c" containerName="maas-api" containerID="cri-o://a4201c91b2fa39993637106a7f88d5c1321524d23052a630755c83191e034a8a" gracePeriod=30 Oct 31 01:29:53.001006 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:53.000972 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:53.125324 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:53.125293 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:53.175004 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:53.174975 4440 generic.go:356] "Generic (PLEG): container finished" podID="747609de-e694-475a-803a-f759ee56c30c" containerID="a4201c91b2fa39993637106a7f88d5c1321524d23052a630755c83191e034a8a" exitCode=0 Oct 31 01:29:53.175162 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:53.175006 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cec774-74f45f5789-2nbrx" event={"ID":"747609de-e694-475a-803a-f759ee56c30c","Type":"ContainerDied","Data":"a4201c91b2fa39993637106a7f88d5c1321524d23052a630755c83191e034a8a"} Oct 31 01:29:53.205506 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:53.200660 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cec774-74f45f5789-2nbrx" Oct 31 01:29:53.242721 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:53.242685 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:53.255898 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:53.255869 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/747609de-e694-475a-803a-f759ee56c30c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/747609de-e694-475a-803a-f759ee56c30c-maas-api-tls\") pod \"747609de-e694-475a-803a-f759ee56c30c\" (UID: \"747609de-e694-475a-803a-f759ee56c30c\") " Oct 31 01:29:53.256050 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:53.255970 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/747609de-e694-475a-803a-f759ee56c30c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/747609de-e694-475a-803a-f759ee56c30c-openshift-service-ca\") pod \"747609de-e694-475a-803a-f759ee56c30c\" (UID: \"747609de-e694-475a-803a-f759ee56c30c\") " Oct 31 01:29:53.256050 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:53.256039 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/747609de-e694-475a-803a-f759ee56c30c-kube-api-access-pqcd2\" (UniqueName: \"kubernetes.io/projected/747609de-e694-475a-803a-f759ee56c30c-kube-api-access-pqcd2\") pod \"747609de-e694-475a-803a-f759ee56c30c\" (UID: \"747609de-e694-475a-803a-f759ee56c30c\") " Oct 31 01:29:53.256153 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:53.256084 4440 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/747609de-e694-475a-803a-f759ee56c30c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/747609de-e694-475a-803a-f759ee56c30c-metrics-certs\") pod \"747609de-e694-475a-803a-f759ee56c30c\" (UID: \"747609de-e694-475a-803a-f759ee56c30c\") " Oct 31 01:29:53.256389 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:53.256357 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/747609de-e694-475a-803a-f759ee56c30c-openshift-service-ca" pod "747609de-e694-475a-803a-f759ee56c30c" (UID: "747609de-e694-475a-803a-f759ee56c30c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:29:53.256577 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:53.256524 4440 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/747609de-e694-475a-803a-f759ee56c30c-openshift-service-ca\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:53.258449 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:53.258367 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/747609de-e694-475a-803a-f759ee56c30c-metrics-certs" pod "747609de-e694-475a-803a-f759ee56c30c" (UID: "747609de-e694-475a-803a-f759ee56c30c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:29:53.258581 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:53.258544 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/747609de-e694-475a-803a-f759ee56c30c-maas-api-tls" pod "747609de-e694-475a-803a-f759ee56c30c" (UID: "747609de-e694-475a-803a-f759ee56c30c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:29:53.258581 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:53.258570 4440 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/747609de-e694-475a-803a-f759ee56c30c-kube-api-access-pqcd2" pod "747609de-e694-475a-803a-f759ee56c30c" (UID: "747609de-e694-475a-803a-f759ee56c30c"). InnerVolumeSpecName "kube-api-access-pqcd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:29:53.357589 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:53.357555 4440 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/747609de-e694-475a-803a-f759ee56c30c-metrics-certs\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:53.357589 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:53.357583 4440 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/747609de-e694-475a-803a-f759ee56c30c-maas-api-tls\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:53.357589 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:53.357593 4440 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqcd2\" (UniqueName: \"kubernetes.io/projected/747609de-e694-475a-803a-f759ee56c30c-kube-api-access-pqcd2\") on node \"ip-10-0-130-39.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:53.661183 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:53.661100 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:54.182275 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:54.182243 4440 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cec774-74f45f5789-2nbrx" Oct 31 01:29:54.182752 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:54.182279 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cec774-74f45f5789-2nbrx" event={"ID":"747609de-e694-475a-803a-f759ee56c30c","Type":"ContainerDied","Data":"ba6118326f3c5c445df4a6bab58448c49d42aebb4bf56d08a63e7aed35ebbc7d"} Oct 31 01:29:54.182752 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:54.182341 4440 scope.go:122] "RemoveContainer" containerID="a4201c91b2fa39993637106a7f88d5c1321524d23052a630755c83191e034a8a" Oct 31 01:29:54.199976 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:54.199953 4440 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cec774-74f45f5789-2nbrx"] Oct 31 01:29:54.214802 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:54.208807 4440 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cec774-74f45f5789-2nbrx"] Oct 31 01:29:54.593331 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:54.593254 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:29:55.772819 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:55.772789 4440 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="747609de-e694-475a-803a-f759ee56c30c" path="/var/lib/kubelet/pods/747609de-e694-475a-803a-f759ee56c30c/volumes" Oct 31 01:29:59.428226 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:59.428190 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:59.548127 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:59.548095 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:59.899151 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:29:59.899034 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:30:01.000378 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:30:01.000338 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:30:01.858519 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:30:01.858480 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:30:04.516443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:30:04.507385 4440 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6rjmk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:30:04.516443 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:30:04.507467 4440 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6rjmk" podUID="dfa83ced-0142-4d9a-9706-7edb28fda974" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:30:06.990992 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:30:06.990945 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:30:22.387776 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:30:22.387740 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:30:30.999976 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:30:30.999938 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:30:36.538969 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:30:36.538925 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:30:47.380119 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:30:47.380079 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:30:49.065864 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:30:49.065824 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6x2mj"] Oct 31 01:31:00.189340 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:00.189300 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-569c46f4cf-fzxcn_fa4b150f-8d6f-4ee3-a150-1977819e47d9/manager/0.log" Oct 31 01:31:00.563793 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:00.563707 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-j86rr_d7e25779-b4c1-46ae-954f-55d8f221a802/manager/0.log" Oct 31 01:31:00.999469 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:31:00.999409 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:31:02.608087 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:02.608052 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7fbdfc6954-vfxfr_deb65aa2-dd4b-449e-891b-9bfc3b3f3b07/maas-api/0.log" Oct 31 01:31:05.736952 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:05.736916 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-tmjs6_e7fa1604-b8a5-4dc7-9b70-2f0eacebaf76/manager/0.log" Oct 31 01:31:05.906605 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:05.906566 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-4k268_5d812e97-43dd-4bd1-b689-410fa1bc32c4/manager/0.log" Oct 31 01:31:06.082963 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:06.082882 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-rnh95_1840b081-35b8-47de-bdd9-5000bed2b2e7/kuadrant-console-plugin/0.log" Oct 31 01:31:06.292728 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:06.292685 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-2j5x4_189472bb-44c7-4a81-bff1-49b621da9b02/manager/0.log" Oct 31 01:31:06.511411 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:06.511379 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-6x2mj_09059326-b484-4ed1-b50f-a4b0d9dfbb44/limitador/0.log" Oct 31 01:31:06.693861 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:06.693824 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-qw7v5_4d6ca668-5b11-4b95-a803-00694d4522e2/manager/0.log" Oct 31 01:31:07.430821 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:07.430790 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68d996c557t8mkr_0c5e03e2-611d-4ef6-804d-a769a4148613/istio-proxy/0.log" Oct 31 01:31:08.210684 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:08.210645 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-6rjmk_dfa83ced-0142-4d9a-9706-7edb28fda974/istio-proxy/1.log" Oct 31 01:31:08.408927 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:08.408887 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-8tjdw_56fbc0ea-4056-4f36-867b-68601554c6fb/istio-proxy/0.log" Oct 31 01:31:08.923497 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:08.923465 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-qhwdc_103d9e31-f559-4366-98ab-2c80cb627428/istio-proxy/0.log" Oct 31 01:31:09.703177 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:09.703145 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-wk87j_57c830db-af34-4fb2-b933-f85f04afb3c3/istio-proxy/0.log" Oct 31 01:31:10.074899 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:10.074823 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8_0bab2f8b-1710-4327-a368-132c18e7bef9/istio-proxy/0.log" Oct 31 01:31:10.305770 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:10.305742 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-dd7nw_c3e485e2-8c87-4e5d-bd9f-639702171908/payload-pre-processing/0.log" Oct 31 01:31:10.489307 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:10.489275 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-w2ptj_e945ee20-e9d8-45e5-9bd3-1fada7c3ca2e/payload-processing/0.log" Oct 31 01:31:11.130720 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:11.130690 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-6rjmk_dfa83ced-0142-4d9a-9706-7edb28fda974/istio-proxy/0.log" Oct 31 01:31:11.855661 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:11.855631 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-kr4hj_76e1f133-7f40-4472-a3f3-c1b6944818d1/main/0.log" Oct 31 01:31:12.040289 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:12.040259 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-tvltr_c14a3342-f1ee-4856-97ff-4c0cc1e25e19/main/0.log" Oct 31 01:31:12.550191 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:12.550165 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-n5jxl_3d2a81a4-1c2e-4e26-8a8a-749d837e1ac1/main/0.log" Oct 31 01:31:12.879874 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:12.879784 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd_2bde1a17-6626-433d-a82c-3cff6eac4a1e/main/0.log" Oct 31 01:31:22.901605 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:22.901571 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-569c46f4cf-fzxcn_fa4b150f-8d6f-4ee3-a150-1977819e47d9/manager/0.log" Oct 31 01:31:23.291254 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:23.291215 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-j86rr_d7e25779-b4c1-46ae-954f-55d8f221a802/manager/0.log" Oct 31 01:31:25.028053 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:25.028022 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7fbdfc6954-vfxfr_deb65aa2-dd4b-449e-891b-9bfc3b3f3b07/maas-api/0.log" Oct 31 01:31:27.731667 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:27.731635 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-tmjs6_e7fa1604-b8a5-4dc7-9b70-2f0eacebaf76/manager/0.log" Oct 31 01:31:27.931538 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:27.931502 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-4k268_5d812e97-43dd-4bd1-b689-410fa1bc32c4/manager/0.log" Oct 31 01:31:28.099596 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:28.099524 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-rnh95_1840b081-35b8-47de-bdd9-5000bed2b2e7/kuadrant-console-plugin/0.log" Oct 31 01:31:28.296118 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:28.296084 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-2j5x4_189472bb-44c7-4a81-bff1-49b621da9b02/manager/0.log" Oct 31 01:31:28.479665 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:28.479633 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-6x2mj_09059326-b484-4ed1-b50f-a4b0d9dfbb44/limitador/0.log" Oct 31 01:31:28.661675 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:28.661641 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-qw7v5_4d6ca668-5b11-4b95-a803-00694d4522e2/manager/0.log" Oct 31 01:31:29.240385 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:29.240350 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68d996c557t8mkr_0c5e03e2-611d-4ef6-804d-a769a4148613/istio-proxy/0.log" Oct 31 01:31:29.982236 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:29.982205 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-6rjmk_dfa83ced-0142-4d9a-9706-7edb28fda974/istio-proxy/1.log" Oct 31 01:31:30.164004 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:30.163971 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-8tjdw_56fbc0ea-4056-4f36-867b-68601554c6fb/istio-proxy/0.log" Oct 31 01:31:30.711374 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:30.711341 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-qhwdc_103d9e31-f559-4366-98ab-2c80cb627428/istio-proxy/0.log" Oct 31 01:31:31.001772 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:31:31.001686 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:31:31.447830 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:31.447789 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-wk87j_57c830db-af34-4fb2-b933-f85f04afb3c3/istio-proxy/0.log" Oct 31 01:31:31.807391 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:31.807303 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8_0bab2f8b-1710-4327-a368-132c18e7bef9/istio-proxy/0.log" Oct 31 01:31:31.972299 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:31.972266 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-dd7nw_c3e485e2-8c87-4e5d-bd9f-639702171908/payload-pre-processing/0.log" Oct 31 01:31:32.151694 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:32.151603 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-w2ptj_e945ee20-e9d8-45e5-9bd3-1fada7c3ca2e/payload-processing/0.log" Oct 31 01:31:33.057236 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:33.057201 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-kr4hj_76e1f133-7f40-4472-a3f3-c1b6944818d1/main/0.log" Oct 31 01:31:33.231130 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:33.231099 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-tvltr_c14a3342-f1ee-4856-97ff-4c0cc1e25e19/main/0.log" Oct 31 01:31:33.778813 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:33.778784 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-n5jxl_3d2a81a4-1c2e-4e26-8a8a-749d837e1ac1/main/0.log" Oct 31 01:31:34.134478 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:34.134384 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-5z5xd_2bde1a17-6626-433d-a82c-3cff6eac4a1e/main/0.log" Oct 31 01:31:47.468470 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:31:47.468428 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:32:01.000651 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:32:01.000608 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:32:13.325738 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:32:13.325699 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-w2ptj_e945ee20-e9d8-45e5-9bd3-1fada7c3ca2e/payload-processing/0.log" Oct 31 01:32:13.499921 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:32:13.499883 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-w2ptj_e945ee20-e9d8-45e5-9bd3-1fada7c3ca2e/payload-processing/0.log" Oct 31 01:32:23.824261 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:32:23.824225 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:32:31.001429 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:32:31.001375 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:32:44.783159 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:32:44.783065 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:32:47.543067 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:32:47.543026 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:33:00.998847 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:33:00.998811 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:33:01.211851 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:01.211811 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-w2ptj_e945ee20-e9d8-45e5-9bd3-1fada7c3ca2e/payload-processing/0.log" Oct 31 01:33:01.388176 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:01.388089 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-w2ptj_e945ee20-e9d8-45e5-9bd3-1fada7c3ca2e/payload-processing/0.log" Oct 31 01:33:01.768801 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:01.768766 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:33:01.918867 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:01.918823 4440 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" limit=1024 Oct 31 01:33:05.253814 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:05.253779 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-s9tlh_49f8a438-6b29-4160-98e1-3f26e27fe0af/global-pull-secret-syncer/0.log" Oct 31 01:33:05.446620 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:05.446575 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8kln5_12002bf6-fce5-4d20-a62c-6782522acd47/konnectivity-agent/0.log" Oct 31 01:33:05.543919 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:05.543835 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-39.us-east-2.compute.internal_eaafb8f2be56642836cc3cf93864aa97/haproxy/0.log" Oct 31 01:33:12.980042 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:12.980011 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-tmjs6_e7fa1604-b8a5-4dc7-9b70-2f0eacebaf76/manager/0.log" Oct 31 01:33:13.018679 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:13.018651 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-4k268_5d812e97-43dd-4bd1-b689-410fa1bc32c4/manager/0.log" Oct 31 01:33:13.055394 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:13.055371 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-rnh95_1840b081-35b8-47de-bdd9-5000bed2b2e7/kuadrant-console-plugin/0.log" Oct 31 01:33:13.130088 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:13.130063 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-2j5x4_189472bb-44c7-4a81-bff1-49b621da9b02/manager/0.log" Oct 31 01:33:13.154920 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:13.154896 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-6x2mj_09059326-b484-4ed1-b50f-a4b0d9dfbb44/limitador/0.log" Oct 31 01:33:13.246304 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:13.246237 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-qw7v5_4d6ca668-5b11-4b95-a803-00694d4522e2/manager/0.log" Oct 31 01:33:14.613333 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:14.613304 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4a15e00b-582f-4f57-90cd-4bf986e2b118/alertmanager/0.log" Oct 31 01:33:14.637754 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:14.637727 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4a15e00b-582f-4f57-90cd-4bf986e2b118/config-reloader/0.log" Oct 31 01:33:14.661742 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:14.661722 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4a15e00b-582f-4f57-90cd-4bf986e2b118/kube-rbac-proxy-web/0.log" Oct 31 01:33:14.685360 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:14.685336 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4a15e00b-582f-4f57-90cd-4bf986e2b118/kube-rbac-proxy/0.log" Oct 31 01:33:14.711512 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:14.711491 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4a15e00b-582f-4f57-90cd-4bf986e2b118/kube-rbac-proxy-metric/0.log" Oct 31 01:33:14.735319 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:14.735300 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4a15e00b-582f-4f57-90cd-4bf986e2b118/prom-label-proxy/0.log" Oct 31 01:33:14.761152 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:14.761135 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4a15e00b-582f-4f57-90cd-4bf986e2b118/init-config-reloader/0.log" Oct 31 01:33:14.996638 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:14.996612 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6qp5m_dd841d54-1f08-42b8-ac54-4d1aac4f81e8/node-exporter/0.log" Oct 31 01:33:15.018383 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:15.018354 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6qp5m_dd841d54-1f08-42b8-ac54-4d1aac4f81e8/kube-rbac-proxy/0.log" Oct 31 01:33:15.041840 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:15.041817 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6qp5m_dd841d54-1f08-42b8-ac54-4d1aac4f81e8/init-textfile/0.log" Oct 31 01:33:15.712980 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:15.712948 4440 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fsd8j/perf-node-gather-daemonset-v26xd"] Oct 31 01:33:15.713367 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:15.713354 4440 cpu_manager.go:414] "containerMap: removing container" podUID="747609de-e694-475a-803a-f759ee56c30c" containerName="maas-api" Oct 31 01:33:15.713433 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:15.713367 4440 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="747609de-e694-475a-803a-f759ee56c30c" containerName="maas-api" Oct 31 01:33:15.713474 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:15.713457 4440 memory_manager.go:365] "RemoveStaleState removing state" podUID="747609de-e694-475a-803a-f759ee56c30c" containerName="maas-api" Oct 31 01:33:15.721561 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:15.718521 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fsd8j/perf-node-gather-daemonset-v26xd" Oct 31 01:33:15.728977 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:15.728949 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fsd8j/perf-node-gather-daemonset-v26xd"] Oct 31 01:33:15.732616 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:15.732595 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28501ca1-833e-42d2-95d5-00d7dde6e8cd-sys\") pod \"perf-node-gather-daemonset-v26xd\" (UID: \"28501ca1-833e-42d2-95d5-00d7dde6e8cd\") " pod="openshift-must-gather-fsd8j/perf-node-gather-daemonset-v26xd" Oct 31 01:33:15.732760 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:15.732622 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/28501ca1-833e-42d2-95d5-00d7dde6e8cd-proc\") pod \"perf-node-gather-daemonset-v26xd\" (UID: \"28501ca1-833e-42d2-95d5-00d7dde6e8cd\") " pod="openshift-must-gather-fsd8j/perf-node-gather-daemonset-v26xd" Oct 31 01:33:15.732760 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:15.732655 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwqwt\" (UniqueName: \"kubernetes.io/projected/28501ca1-833e-42d2-95d5-00d7dde6e8cd-kube-api-access-bwqwt\") pod \"perf-node-gather-daemonset-v26xd\" (UID: \"28501ca1-833e-42d2-95d5-00d7dde6e8cd\") " pod="openshift-must-gather-fsd8j/perf-node-gather-daemonset-v26xd" Oct 31 01:33:15.732760 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:15.732739 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28501ca1-833e-42d2-95d5-00d7dde6e8cd-lib-modules\") pod \"perf-node-gather-daemonset-v26xd\" (UID: \"28501ca1-833e-42d2-95d5-00d7dde6e8cd\") " pod="openshift-must-gather-fsd8j/perf-node-gather-daemonset-v26xd" Oct 31 01:33:15.732929 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:15.732809 4440 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/28501ca1-833e-42d2-95d5-00d7dde6e8cd-podres\") pod \"perf-node-gather-daemonset-v26xd\" (UID: \"28501ca1-833e-42d2-95d5-00d7dde6e8cd\") " pod="openshift-must-gather-fsd8j/perf-node-gather-daemonset-v26xd" Oct 31 01:33:15.810234 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:15.810200 4440 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fsd8j\"/\"default-dockercfg-mphnn\"" Oct 31 01:33:15.815667 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:15.815646 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fsd8j\"/\"openshift-service-ca.crt\"" Oct 31 01:33:15.815929 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:15.815912 4440 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fsd8j\"/\"kube-root-ca.crt\"" Oct 31 01:33:15.833451 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:15.833403 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28501ca1-833e-42d2-95d5-00d7dde6e8cd-sys\") pod \"perf-node-gather-daemonset-v26xd\" (UID: \"28501ca1-833e-42d2-95d5-00d7dde6e8cd\") " pod="openshift-must-gather-fsd8j/perf-node-gather-daemonset-v26xd" Oct 31 01:33:15.833542 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:15.833454 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/28501ca1-833e-42d2-95d5-00d7dde6e8cd-proc\") pod \"perf-node-gather-daemonset-v26xd\" (UID: \"28501ca1-833e-42d2-95d5-00d7dde6e8cd\") " pod="openshift-must-gather-fsd8j/perf-node-gather-daemonset-v26xd" Oct 31 01:33:15.833542 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:15.833491 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwqwt\" (UniqueName: \"kubernetes.io/projected/28501ca1-833e-42d2-95d5-00d7dde6e8cd-kube-api-access-bwqwt\") pod \"perf-node-gather-daemonset-v26xd\" (UID: \"28501ca1-833e-42d2-95d5-00d7dde6e8cd\") " pod="openshift-must-gather-fsd8j/perf-node-gather-daemonset-v26xd" Oct 31 01:33:15.833542 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:15.833509 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28501ca1-833e-42d2-95d5-00d7dde6e8cd-lib-modules\") pod \"perf-node-gather-daemonset-v26xd\" (UID: \"28501ca1-833e-42d2-95d5-00d7dde6e8cd\") " pod="openshift-must-gather-fsd8j/perf-node-gather-daemonset-v26xd" Oct 31 01:33:15.833542 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:15.833521 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28501ca1-833e-42d2-95d5-00d7dde6e8cd-sys\") pod \"perf-node-gather-daemonset-v26xd\" (UID: \"28501ca1-833e-42d2-95d5-00d7dde6e8cd\") " pod="openshift-must-gather-fsd8j/perf-node-gather-daemonset-v26xd" Oct 31 01:33:15.833542 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:15.833539 4440 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/28501ca1-833e-42d2-95d5-00d7dde6e8cd-podres\") pod \"perf-node-gather-daemonset-v26xd\" (UID: \"28501ca1-833e-42d2-95d5-00d7dde6e8cd\") " pod="openshift-must-gather-fsd8j/perf-node-gather-daemonset-v26xd" Oct 31 01:33:15.833753 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:15.833578 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/28501ca1-833e-42d2-95d5-00d7dde6e8cd-proc\") pod \"perf-node-gather-daemonset-v26xd\" (UID: \"28501ca1-833e-42d2-95d5-00d7dde6e8cd\") " pod="openshift-must-gather-fsd8j/perf-node-gather-daemonset-v26xd" Oct 31 01:33:15.833753 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:15.833591 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/28501ca1-833e-42d2-95d5-00d7dde6e8cd-podres\") pod \"perf-node-gather-daemonset-v26xd\" (UID: \"28501ca1-833e-42d2-95d5-00d7dde6e8cd\") " pod="openshift-must-gather-fsd8j/perf-node-gather-daemonset-v26xd" Oct 31 01:33:15.833753 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:15.833610 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28501ca1-833e-42d2-95d5-00d7dde6e8cd-lib-modules\") pod \"perf-node-gather-daemonset-v26xd\" (UID: \"28501ca1-833e-42d2-95d5-00d7dde6e8cd\") " pod="openshift-must-gather-fsd8j/perf-node-gather-daemonset-v26xd" Oct 31 01:33:15.841650 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:15.841631 4440 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwqwt\" (UniqueName: \"kubernetes.io/projected/28501ca1-833e-42d2-95d5-00d7dde6e8cd-kube-api-access-bwqwt\") pod \"perf-node-gather-daemonset-v26xd\" (UID: \"28501ca1-833e-42d2-95d5-00d7dde6e8cd\") " pod="openshift-must-gather-fsd8j/perf-node-gather-daemonset-v26xd" Oct 31 01:33:16.038486 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:16.038383 4440 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fsd8j/perf-node-gather-daemonset-v26xd" Oct 31 01:33:16.312987 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:16.312959 4440 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fsd8j/perf-node-gather-daemonset-v26xd"] Oct 31 01:33:16.314282 ip-10-0-130-39 kubenswrapper[4440]: W1031 01:33:16.314249 4440 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod28501ca1_833e_42d2_95d5_00d7dde6e8cd.slice/crio-93b9d86891ae666e4c022f221290a259792c7397d7efbd71af132fec04641357 WatchSource:0}: Error finding container 93b9d86891ae666e4c022f221290a259792c7397d7efbd71af132fec04641357: Status 404 returned error can't find the container with id 93b9d86891ae666e4c022f221290a259792c7397d7efbd71af132fec04641357 Oct 31 01:33:16.419451 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:16.419410 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fsd8j/perf-node-gather-daemonset-v26xd" event={"ID":"28501ca1-833e-42d2-95d5-00d7dde6e8cd","Type":"ContainerStarted","Data":"e235a7ece8598c454c535849d1564fed20468d7fad7c082ecf11669cc831c409"} Oct 31 01:33:16.419547 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:16.419477 4440 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fsd8j/perf-node-gather-daemonset-v26xd" event={"ID":"28501ca1-833e-42d2-95d5-00d7dde6e8cd","Type":"ContainerStarted","Data":"93b9d86891ae666e4c022f221290a259792c7397d7efbd71af132fec04641357"} Oct 31 01:33:16.419547 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:16.419517 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fsd8j/perf-node-gather-daemonset-v26xd" Oct 31 01:33:16.436942 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:16.436897 4440 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-fsd8j/perf-node-gather-daemonset-v26xd" podStartSLOduration=1.436882595 podStartE2EDuration="1.436882595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:33:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:33:16.435613947 +0000 UTC m=+2240.794336266" watchObservedRunningTime="2026-10-31 01:33:16.436882595 +0000 UTC m=+2240.795604915" Oct 31 01:33:18.432671 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:18.432642 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-gjjhv_6ed6c333-706a-44dc-aae6-54be82855800/download-server/0.log" Oct 31 01:33:20.449411 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:20.449382 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zhmhj_49f7c8f4-b979-436d-b035-20f9fcae140a/dns/0.log" Oct 31 01:33:20.472817 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:20.472793 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zhmhj_49f7c8f4-b979-436d-b035-20f9fcae140a/kube-rbac-proxy/0.log" Oct 31 01:33:20.528003 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:20.527979 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qsppl_cd654994-98a0-4d39-aa38-270b4a0d4341/dns-node-resolver/0.log" Oct 31 01:33:21.155958 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:21.155913 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-85c74bc6cd-ggkcd_faccb6fb-036d-4190-a0e8-88f417c6a29b/registry/0.log" Oct 31 01:33:21.236098 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:21.236071 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qzm2t_0a37f05b-0755-4ca7-8d9b-12030234bec1/node-ca/0.log" Oct 31 01:33:22.440902 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:22.440875 4440 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fsd8j/perf-node-gather-daemonset-v26xd" Oct 31 01:33:22.453035 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:22.453009 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68d996c557t8mkr_0c5e03e2-611d-4ef6-804d-a769a4148613/istio-proxy/0.log" Oct 31 01:33:24.031082 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:24.031012 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-6rjmk_dfa83ced-0142-4d9a-9706-7edb28fda974/istio-proxy/0.log" Oct 31 01:33:24.033387 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:24.033360 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-6rjmk_dfa83ced-0142-4d9a-9706-7edb28fda974/istio-proxy/1.log" Oct 31 01:33:24.062864 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:24.062841 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-8tjdw_56fbc0ea-4056-4f36-867b-68601554c6fb/istio-proxy/0.log" Oct 31 01:33:24.190735 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:24.190704 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-qhwdc_103d9e31-f559-4366-98ab-2c80cb627428/istio-proxy/0.log" Oct 31 01:33:24.326650 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:24.326571 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-wk87j_57c830db-af34-4fb2-b933-f85f04afb3c3/istio-proxy/0.log" Oct 31 01:33:24.401112 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:24.401074 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8_0bab2f8b-1710-4327-a368-132c18e7bef9/istio-proxy/0.log" Oct 31 01:33:24.430951 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:24.430928 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-dd7nw_c3e485e2-8c87-4e5d-bd9f-639702171908/payload-pre-processing/0.log" Oct 31 01:33:24.459262 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:24.459240 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-w2ptj_e945ee20-e9d8-45e5-9bd3-1fada7c3ca2e/payload-processing/0.log" Oct 31 01:33:26.237025 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:26.236992 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-srw8h_9a11b541-6b88-439e-bcbe-5901cb4c20b0/serve-healthcheck-canary/0.log" Oct 31 01:33:26.960262 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:26.960226 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lglrb_a2b4a048-eaac-436d-8439-61a585d03601/kube-rbac-proxy/0.log" Oct 31 01:33:26.987717 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:26.987682 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lglrb_a2b4a048-eaac-436d-8439-61a585d03601/exporter/0.log" Oct 31 01:33:27.011365 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:27.011337 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lglrb_a2b4a048-eaac-436d-8439-61a585d03601/extractor/0.log" Oct 31 01:33:27.084834 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:27.084808 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68d996c557t8mkr_0c5e03e2-611d-4ef6-804d-a769a4148613/istio-proxy/0.log" Oct 31 01:33:28.618164 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:28.618128 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-6rjmk_dfa83ced-0142-4d9a-9706-7edb28fda974/istio-proxy/0.log" Oct 31 01:33:28.620359 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:28.620335 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-6rjmk_dfa83ced-0142-4d9a-9706-7edb28fda974/istio-proxy/1.log" Oct 31 01:33:28.636899 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:28.636879 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-8tjdw_56fbc0ea-4056-4f36-867b-68601554c6fb/istio-proxy/0.log" Oct 31 01:33:28.689917 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:28.689889 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-qhwdc_103d9e31-f559-4366-98ab-2c80cb627428/istio-proxy/0.log" Oct 31 01:33:28.763708 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:28.763678 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-wk87j_57c830db-af34-4fb2-b933-f85f04afb3c3/istio-proxy/0.log" Oct 31 01:33:28.799246 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:28.799221 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8_0bab2f8b-1710-4327-a368-132c18e7bef9/istio-proxy/0.log" Oct 31 01:33:28.819342 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:28.819325 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-dd7nw_c3e485e2-8c87-4e5d-bd9f-639702171908/payload-pre-processing/0.log" Oct 31 01:33:28.835283 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:28.835263 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-w2ptj_e945ee20-e9d8-45e5-9bd3-1fada7c3ca2e/payload-processing/0.log" Oct 31 01:33:29.856254 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:29.856221 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-7f58f8c74c-gl94n_513526cf-d6aa-47fe-a8a8-e2191cf2be3d/manager/0.log" Oct 31 01:33:29.874338 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:29.874315 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-7f58f8c74c-gl94n_513526cf-d6aa-47fe-a8a8-e2191cf2be3d/copy-manifests/0.log" Oct 31 01:33:30.832086 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:30.832053 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68d996c557t8mkr_0c5e03e2-611d-4ef6-804d-a769a4148613/istio-proxy/0.log" Oct 31 01:33:30.999616 ip-10-0-130-39 kubenswrapper[4440]: E1031 01:33:30.999582 4440 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lglrb" containerName="extractor" Oct 31 01:33:32.132568 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:32.132537 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-569c46f4cf-fzxcn_fa4b150f-8d6f-4ee3-a150-1977819e47d9/manager/0.log" Oct 31 01:33:32.233980 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:32.233950 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-j86rr_d7e25779-b4c1-46ae-954f-55d8f221a802/manager/0.log" Oct 31 01:33:32.432861 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:32.432836 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-6rjmk_dfa83ced-0142-4d9a-9706-7edb28fda974/istio-proxy/0.log" Oct 31 01:33:32.434854 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:32.434799 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-6rjmk_dfa83ced-0142-4d9a-9706-7edb28fda974/istio-proxy/1.log" Oct 31 01:33:32.466282 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:32.466258 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-8tjdw_56fbc0ea-4056-4f36-867b-68601554c6fb/istio-proxy/0.log" Oct 31 01:33:32.550922 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:32.550893 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-qhwdc_103d9e31-f559-4366-98ab-2c80cb627428/istio-proxy/0.log" Oct 31 01:33:32.642015 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:32.641991 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-wk87j_57c830db-af34-4fb2-b933-f85f04afb3c3/istio-proxy/0.log" Oct 31 01:33:32.676060 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:32.676034 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-9b4747bc-4x2q8_0bab2f8b-1710-4327-a368-132c18e7bef9/istio-proxy/0.log" Oct 31 01:33:32.694387 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:32.694267 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-dd7nw_c3e485e2-8c87-4e5d-bd9f-639702171908/payload-pre-processing/0.log" Oct 31 01:33:32.711726 ip-10-0-130-39 kubenswrapper[4440]: I1031 01:33:32.711701 4440 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-w2ptj_e945ee20-e9d8-45e5-9bd3-1fada7c3ca2e/payload-processing/0.log"